Evaluation designs seek to provide evidence of impact and collect formative data to understand and improve programs and inform decisions. Formative data is commonly used at the onset of developing a new program to modify and improve before full implementation. However, formative data is often collected throughout the life of the program to shape…

Introduction: Missing data are bound to occur in education intervention research. Reasons vary but always have the consequence of reducing sample sizes and can, under certain conditions, seriously bias estimated intervention effects. A wide array of methods have been developed for handling missing data (Enders, 2023). Whereas older approaches…

We consider a class of multiple-group individually-randomized group trials (IRGTs) that introduces a (partially) cross-classified structure in the treatment condition (only). The novel feature of this design is that the nature of the treatment induces a clustering structure that involves two or more non-nested groups among individuals in the…

Background: In educational settings, individuals are often best served by an intervention that is adapted over sequential stages to suit their initial and changing needs. The salience of an adaptive intervention is, perhaps, most clear in the classroom. Learning itself is a sequential process: mastering a given concept or technique frequently…

Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

I. Purpose of the Study: Detecting whether interventions work or not (through main effect analysis) can provide empirical evidence regarding the causal linkage between malleable factors (e.g., interventions) and learner outcomes. In complement, moderation analyses help delineate for whom and under what conditions intervention effects are most…

Background: Evidence from education randomized controlled trials (RCTs) in low- and middle-income countries (LMICs) demonstrates how interventions can improve children's educational achievement [1, 2, 3, 4]. RCTs assess the impact of an intervention by comparing outcomes--aggregate test scores--between treatment and control groups. A review of…

Background: The analysis of mediating mechanisms attempts to examine the impact of, for example, teacher training by dismantling and testing the core components that comprise an underlying theory and program. This process is seen as a critical step in evaluating programs and mapping teacher development because it provides empirical tests of each…

Background: Education scientists are increasingly interested in constructing interventions that are adaptive over time to suit the evolving needs of students, classrooms, or schools. Such "adaptive interventions" (also referred to as dynamic treatment regimens or dynamic instructional regimes) determine which treatment should be offered…

Background and Context: Every student is a powerful learner capable of succeeding in mathematics. The EF+Math Program aims to dramatically improve math outcomes for Black and Latinx students and students of all races experiencing poverty (hereafter, our Priority Students). We are a community of researchers, developers, educators, and students who…

Context: A perennial problem in applied statistics is the inability to justify strong claims about cause-and-effect relationships without full knowledge of the mechanism determining selection into treatment. Few research designs other than the well-implemented random assignment study meet this requirement. Researchers have proposed partial…

Background: Education researchers typically estimate average program effects with regression; if they are interested in heterogeneous effects, they include an interaction in the model. Such models quantify and infer the influences of each covariate on the effect via interaction coefficients and their associated p-values or confidence intervals.…

Why we need better reports of teacher professional development interventions for finding out what works where: Worldwide, billions of dollars are invested yearly in the professional development of in-service teachers ($14 billion in the USA alone, as of 2014; Bill & Melinda Gates Foundation, 2014). This makes the question of what effects TPD…

Background: Regression Discontinuous Design (RDD) is widely used in educational studies. Through RDD, researchers can obtain unbiased results when Randomized Experimental Design (RED) is inaccessible. Compared to RED, the RDD only requires a cut score variable (continuous) and a cutoff value to assign students to the treatment or control groups.…

Background/Context: Methodological background: Meta-analysis typically depends on the assumption that true effects follow the normal distribution. While assuming normality of effect "estimates" is often supported by a central limit theorem, normality for the distribution of interventions' "true" effects is a computational…