Although the COVID-19 pandemic was a major disruptor to K-12 learning and instruction in general, there is particular concern that there was differential impact by student group and instructional mode offered (e.g., in-person versus remote learning). The National Assessment of Educational Progress (NAEP) Long-Term Trend (LTT) assessment administered in 2020 and 2022 to age 9 students (primarily grade 4) offers a unique opportunity to analyze changes in student performance before and during the pandemic: The 2020 timepoint occurred immediately prior to schools closing in March 2020, the majority of schools in 2022 were also sampled in 2020, both years constitute a national sample, a single, summative assessment program was used, and the 2022 school administrator survey included a key question related to instructional modes offered in the 2020-2021 school year. In mathematics, controlling for the school (with school fixed effects) and student background variables, we found students identifying as Black or Hispanic had mean score losses from 2020 to 2022 that were 10 and 4 points, larger than White students, respectively. Similarly, students eligible for free-or-reduced-priced lunch had mean score losses 3 and 4 points larger in mathematics and reading, respectively, than those not eligible, controlling for the same school and other student covariates. In addition, for schools that had at least some classes going remote in the 2020-21 school year, student scores dropped about 7 and 8 more points in mathematics and reading, respectively, on average, from 2020 to 2022 than schools that had no classes going remote, holding all other covariates constant. These findings are critical to understanding variability in educational progress following the COVID-19 pandemic and can support planning for future disasters.