This article offers a case example of how experimental evaluation methods can be coupled with principles of design-based implementation research (DBIR), improvement science (IS), and rapid-cycle evaluation (RCE) methods to provide relatively quick, low-cost, credible assessments of strategies designed to improve programs, policies, or practices.…

In multisite trials, learning about treatment effect variation across sites is critical for understanding where and for whom a program works. Unadjusted comparisons, however, capture "compositional" differences in the distributions of unit-level features as well as "contextual" differences in site-level features, including…

This paper considers recent efforts to conduct experimental and quasi-experimental evaluations of career and technical education programs. It focuses on understanding the counterfactual, or control population, for these program evaluations, discussing how the educational experiences of the control population might vary from those of the treated…

In 2015, Abt Associates received a grant from the Institutes for Education Sciences (IES) for a five-year "Development and Innovation" study of PTC. The purposes of the study were to gauge progress in implementing PTC and to develop and test improvements where needed. Fein et al. (2020) summarize the IES study's approach and findings. A…

This article reviews important statistical methods for estimating the impact of interventions on outcomes in education settings, particularly programs that are implemented in field, rather than laboratory, settings. We begin by describing the causal inference challenge for evaluating program effects. Then four research designs are discussed that…

This case study describes a small randomized controlled trial (RCT) comparing alternative strategies for monitoring and supporting academic achievement in Year Up's Professional Training Corps (PTC) program. Year Up is a nonprofit organization dedicated to preparing economically disadvantaged young adults for well-paying jobs with advancement…

Given the widespread use of non-experimental (NE) methods for assessing program impacts, there is a strong need to know whether NE approaches yield causally valid results in field settings. In within-study comparison (WSC) designs, the researcher compares treatment effects from an NE with those obtained from a randomized experiment that shares the…

In basic regression discontinuity (RD) designs, causal inference is limited to the local area near a single cutoff. To strengthen the generality of the RD treatment estimate, a design with multiple cutoffs along the assignment variable continuum can be applied. The availability of multiple cutoffs allows estimation of a pooled average treatment…

Background and purpose: Studies of education and learning that were described as experiments have been carried out in the USA by educational psychologists since about 1900. In this paper, we discuss the history of randomised trials in education in the USA in terms of five historical periods. In each period, the use of randomised trials was…

Randomized trials are often conducted with separate randomizations across multiple sites such as schools, voting districts, or hospitals. These sites can differ in important ways, including the site's implementation, local conditions, and the composition of individuals. An important question in practice is whether--and under what…

Social-emotional learning (SEL) programs are frequently evaluated using randomized controlled trial (RCT) methodology as a means to assess program impacts. What is often missing in RCT studies is a robust parallel investigation of the multi-level implementation of the program. The field of implementation science bridges the gap between the RCT…

Background: Excessive student anxiety is a common problem that severely impairs short- and long-term academic functioning and increases teacher burden. Reducing student anxiety has been associated with improvement in educational functioning. Because anxiety manifests daily in the classroom, teachers are in an ideal position to identify and help…

This study evaluated the efficacy of the Preschool First Step (PFS) to Success early intervention for children at risk for attention deficit hyperactivity disorder (ADHD). PFS is a targeted intervention for children 3-5 years old with externalizing behavior problems and addresses secondary prevention goals and objectives. As part of a larger…

Data-based instructional programs have proliferated in American schools despite limited evidence of their effectiveness in improving educator practice and raising student achievement. We report results from a two-year school-randomized evaluation of the Achievement Network (ANet), a program providing schools with standards-aligned interim…