Randomized A/B tests allow causal estimation without confounding but are often under-powered. This paper uses a new dataset, including over 250 randomized comparisons conducted in an online learning platform, to illustrate a method combining data from A/B tests with log data from users who were not in the experiment. Inference remains exact and…

Large-scale, classroom-based experiments using adaptive instructional software pose somewhat unique challenges for experimental design and deployment. One reason for this is that adaptive software allows students to advance through the curriculum at different rates and encounter content at different times, meaning that content targeted for…

As online learning platforms become more ubiquitous throughout various curricula, there is a growing need to evaluate the effectiveness of these platforms and the different methods used to structure online education and tutoring. Towards this endeavor, some platforms have performed randomized controlled experiments to compare different user…

Theories regarding the long-term effects of educational interventions are often assumed, but rarely tested using experimental methods. In the following commentary, we argue that the shortage of randomized control trials with long-term follow-up presents serious problems for the field, as it hampers our ability to develop educational programs that…

Adults adopt the traits of characters in narratives, but little is known about whether children do so. In Study 1, 7- and 10-year-olds (N = 96) heard a 2.5-minute recording about a professor or cheerleader. Reporting higher engagement in the professor narrative related to more time playing with an analytical toy (a Rubik's cube), whereas reporting…

Source-based essays are evaluated both on the quality of the writing and the content appropriate interpretation and use of source material. Hence, composing a high-quality source-based essay (an essay written based on source material) relies on skills related to both reading (the sources) and writing (the essay) skills. As such, source-based…

Background: Cluster randomized experiments that assign intact groups such as schools or school districts to treatment conditions are increasingly common in educational research. Such experiments are inherently multilevel designs whose sensitivity (statistical power and precision of estimates) depends on the variance decomposition across levels.…