Background: Automated writing evaluation (AWE) systems, used as formative assessment tools in writing classrooms, are promising for enhancing instruction and improving student performance. Although meta-analytic evidence supports AWE's effectiveness in various contexts, research on its effectiveness in the U.S. K-12 setting has lagged behind its rapid adoption. Further rigorous studies are needed to investigate the effectiveness of AWE within the U.S. K-12 context. Objectives: This study aims to investigate the usage and effectiveness of the Utah Compose AWE system on students' state test English Language Arts (ELA) performance in its first year of statewide implementation. Methods: The sample included all students from grades 4-11 during the school year 2015 in Utah (N = 337,473). Employing a quasi-experimental design using generalised boosted modelling for propensity score weighting, the analysis focused on estimating the average treatment effects among the treated (ATT) of the AWE system. Results and Conclusions: The results showed that students who utilised AWE more frequently demonstrated improved ELA performance compared to their counterparts with lower or no usage. The effects varied across certain student demographic groups. This study provides strong and systematic evidence to support the hypothesis of causal inferences regarding AWE's effects within a large-scale, naturalistic implementation, offering valuable insights for stakeholders seeking to understand the effectiveness of AWE systems.