ERIC Number: ED659487
Record Type: Non-Journal
Publication Date: 2023-Sep-30
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Learning Loss and Recovery during the Coronavirus Pandemic: Evidence from Multiple Districts and States
Sara Hu; Robert Meyer; Michael Christian
Society for Research on Educational Effectiveness
Background: In the immediate aftermath of school closures related to COVID-19, efforts to unpack the impact of the pandemic on student outcomes surged. As schools have returned to fully in-person learning, that focus has shifted towards the implementation and evaluation of interventions to remediate or accelerate impacted learning. However, measuring the ongoing effects of the pandemic on students is crucial for districts and states to be able to monitor when recovery is taking place, and for which students--not only for education agency leaders, who rely on such data to inform the administration of interventions and application of ESSER funds, but also for the broader education field, who need these data to paint an accurate picture of the current state of student achievement and growth. Research Questions: This paper addresses the following six research questions: 1. What was the magnitude of cumulative average learning change, by grade and by state, over each phase of the COVID pandemic: (a) Fall 2019 (or Winter 2019/20) to Fall 2020, (b) Fall 2019 to Winter 2020/21, (c) Fall 2019 to Spring 2021, (d) Fall 2019 to Fall 2021, and (e) Fall 2019 to Winter 2021/22 (with data for this last period available in Winter 2022) 2. Did magnitude of learning change differ for different types of students, particularly along African American students, economically disadvantage students and students with lower prior achievement? 3. What was the magnitude of recovery, if any, from learning losses experienced during the early phases of the pandemic? Are there any heterogeneous effects of recovery along the dimension of race or Data Collection. We utilize a rich historical set of data that includes spring state assessment data and interim assessment data measured in the fall, winter, and spring over multiple years before and after onset of the pandemic. A major strength of the research is that it utilizes assessment data from multiple state assessments and from multiple interim assessments, including NWEA MAP, Renaissance Star, Curriculum Associates i-Ready, and HMH Reading Inventory and Math Inventory. Hence, as mentioned above, the results are expected to provide evidence on whether learning loss is sensitive to the type of assessment used to measure student achievement and to assess the robustness of the results. Methods: This paper uses multi-level achievement growth models (Chetty, Friedman, and Rockoff (2014) and Kane and Staiger (2008)) designed to identify the effects of the covid pandemic on growth in student achievement over the period Fall 2019 to Winter 2021/22, using prior data from school years 2017-18 and 2018-19 as baseline, pre-covid data. The models embody features of evaluation models and standard growth and value-added models. The evaluation lens prompts us to view the COVID pandemic as a type of treatment (or intervention) and the pre-COVID period as a control. Moreover, given that some districts and schools provided instruction at home while other provided instruction in school (or a hybrid blend of the two), it is potentially important to consider these types of instruction as essentially different interventions. Consistent with this view, we address the need to control for possible differences in the treatment interventions and control periods and samples with respect to the composition of students and schools. These considerations are potentially important given that participation rates in interim assessments during the COVID-affected years have differed substantially from participation rates in prior years and the types of schools providing in-home versus in-school instruction likely have differed. Our models allow slope coefficients to vary across years. The model thus allows for the likely possibility that the effects of demographic characteristics and prior achievement may have changed because of the pandemic, with at-risk students experiencing lower growth than in prior years. This model addresses the concern that the expected decline in growth during covid-affected years may have been especially acute for some schools and for some students within schools. Findings: Although we intend to report findings based on the most up-to-date data at the time of the fall conference, our findings based on data through Fall 2021 confirm that learning loss has been substantial for students. However, the evidence suggests that the magnitude of learning loss varied across schools and was highly correlated with factors associated with school and family learning supports. Larger learning losses occurred for students who were enrolled in remote or hybrid learning compared to in-person learners. Conclusions: We develop a model for measuring learning change that estimates separate value-added growth models for comparable periods of time before and after the COVID-19 pandemic, treating the pre-COVID period as a type of control group and the post-COVID period as a type of treatment group. This approach explicitly models different impacts of COVID-19 on students with different characteristics. It can be used to measure learning change both on average and for students in particular demographic groups in a way that is robust to differences in school assignments, demographic makeup, and student achievement between available pre- and post-COVID student samples. Using data from multiple interim assessments in one state, we find evidence of persistent learning lag to Fall 2021, both overall and across students by economic disadvantage and by English language learner status, as shown in figure 1 through figure 3. We are planning on updating results using the latest data which will be available in summer 2023.
Descriptors: Achievement Gains, COVID-19, Pandemics, Age Differences, Instructional Program Divisions, African American Students, Economically Disadvantaged, Low Achievement
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A