NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED656788
Record Type: Non-Journal
Publication Date: 2021-Sep-29
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
A Rapid Evidence Review of Distance Learning Programs
Sarah Sahni; Joshua Polanin; Qi Zhang; Laura Michaelson; Sarah Caverly; Madeline Polese; Ji Hyun Yang
Society for Research on Educational Effectiveness
On March 11, 2020, the World Health Organization declared the spread of COVID-19 a pandemic (World Health Organization, 2020), and the ongoing public health crisis affected all aspects of daily life worldwide. Policies to mitigate the spread of the virus prompted school closures nationwide, disrupting the learning of approximately 50.8 million public school students in the 2019/20 academic year (Education Week, 2020a). To continue to educate American students, educators and school administrators needed to understand the variety of available distance learning models and programs, defined broadly to include any digital educational material designed to be accessed outside the traditional in-person school setting. Existing distance learning programs encompass a wide range of models: fully online schools, individual courses offered entirely online, and hybrid courses that offer a mix of in-person and remote instruction (Digital Learning Collaborative, 2020). In order to quickly and adeptly make use of the available digital programming, educators and administrators need resources to help understand the effectiveness of these programs. They also need information about program characteristics to ensure that programs are suitable for specific settings. Unfortunately, educators and administrators do not have access to this critical information on characteristics of distance learning programs, their quality and effectiveness, and the supporting evidence (Tosh, Doan, Woo, & Henry, 2020). To fill this need, we conducted a rapid evidence review of distance learning programs, following rigorous methodology outlined by the What Works Clearinghouse (WWC) to ensure that decisionmakers received evidence derived from high-quality program evaluations (WWC, 2020a, 2020b). Research Questions: 1. How many distance learning programs have been evaluated in studies that Met WWC Group Design Standards? What are the characteristics of these studies (such as design, sample, and outcomes) and the programs' components (such as program type, content coverage, and teacher interaction style)? 2. Where is evidence needed to address gaps in the distance learning literature? 3. What is the overall average effect of distance learning programs that meet WWC standards on English language arts (ELA) and mathematics achievement? 4. Do any of the programs' components explain differences in effectiveness? Included Studies: We used two strategies to identify potential studies for the rapid evidence review: 1) A broad call to education researchers, practitioners, developers, and other stakeholders to submit studies that evaluated distance learning programs; and 2) A traditional online database search of the Education Resources Information Center. The inclusion criteria followed the WWC's Review of Individual Studies Protocol version 4.0 to guide eligibility decisions regarding the sample, location, outcome measure, and outcome domain. Additionally, because the goal of this effort was to provide practitioners with the most rigorous evidence on effective programs, several additional criteria were specified at the outset of the systematic review process including: Intervention type. Educational programming for students in grades K-16 that was fully online or could be offered completely at a distance. Study design. The study must have used an RCT design and be peer-reviewed. Significant finding. The study must have at least one positive statistically significant finding in an eligible outcome domain. Date of publication. Published during or after 2010. Research Design and Analysis: The literature search and nomination processes yielded 932 studies that were screened according to the inclusion criteria, resulting in 36 studies potentially eligible for WWC Group Design Standards. These studies were reviewed under WWC Group Design Standards version 2.1 or later. Additionally, the project team extracted a set of program components. The components included whether the program provided a complete course or was supplemental, whether it covered broad or narrow content, the type of interaction (teacher-led synchronous, teacher-led asynchronous, or automated/not teacher directed), whether the program was personalized or differentiated based on student performance (adaptive or nonadaptive), and whether the program was gamified (had goals, and followed a story arc). Eligible studies were used to create an evidence gap map. Studies that "Met WWC Group Design Standards" were synthesized using the WWC's preferred approach: a fixed effects, meta-analytic model, weighting the effect sizes by the inverse of the effect size variance (WWC, 2020b). Moderator analyses of the program components were also conducted. Results: Fifteen studies "Met WWC Group Design Standards"; of those, three met the Every Student Succeeds Act Tier 1 requirements. An analysis of where research has been conducted revealed several gaps in the evidence base (see Figure 1). Several distance learning programs for K-8 students "Met WWC Group Design Standards," but only one study of a distance learning program for high school students "Met WWC Group Design Standards." We also see a lack of studies that examine social-emotional outcomes. This is especially notable, as one concern with distance learning is the social-emotional health of the students (Education Week, 2020b; Walker, 2020). In addition, a meta-analysis of studies with similar design characteristics (nine in total) found that, on average, students in the distance learning programs improved in the English language arts outcome domain but not in the mathematics domain. Moderator analyses were conducted to examine whether program components explained differences in effectiveness but did not yield any statistically significant findings. Although the results are promising, continued research using rigorous, randomized designs should be a priority. Conclusions: The sharp transition to remote education in March of 2020 created enormous pressure on administrators to provide adequate resources and distance learning programming to students. Now that education stakeholders have experience implementing these models, regardless of the global pandemic's longevity, distance learning may continue to be used by educators for a variety of reasons. This review revealed several distance learning products that education stakeholders should consider, depending on their situation. Equally important, it found that distance learning currently requires much greater investment, study, and rigorous evaluation. Through increased resources and a greater understanding of the existing evidence on effective distance learning programs, the WWC may be able to decrease the burden placed on education practitioners as they grapple with an instructional landscape that is shifting because of the coronavirus pandemic and may help them to better support their students now and in the future.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Information Analyses
Education Level: Elementary Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A