Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

Conceptual replications have received increased coverage in the educational research agenda. This article argues for clarity in, and justification of, the definition, scope, and boundaries of a conceptual replication and what it can and cannot do. It argues for clear justifications when changing components from those of the original study. The…

Experimental studies of educational interventions are rarely based on representative samples of the target population. This simulation study tests two formal sampling strategies for selecting districts and schools from within strata when they may not agree to participate if selected: (1) balanced selection of the most typical district or school…

Students often believe that statistical significance is the only determinant of whether a quantitative result is "important." In this paper, I review traditional null hypothesis statistical testing to identify what questions inferential statistics can and cannot answer, including statistical significance, effect size and direction,…

Findings from experimental studies have increasingly been used to inform policy in school settings. Thus far, the populations in many of these studies are typically defined in a cross-sectional context; namely, the populations are defined in the same academic year in which the study took place or the population is defined at a fixed time point.…