In the complex world of educational policy, causal inferences will be debated. As we review non-experimental designs in educational policy, we focus on how to clarify and focus the terms of debate. We begin by presenting the potential outcomes/counterfactual framework and then describe approximations to the counterfactual generated from the…

This article discusses estimation of average treatment effects for randomized controlled trials (RCTs) using grouped administrative data to help improve data access. The focus is on design-based estimators, derived using the building blocks of experiments, that are conducive to grouped data for a wide range of RCT designs, including clustered and…

Despite ongoing disagreement about what kinds of evidence are most valuable to education, there continues to be an increasing push to make a narrow group of scientific methodologies the basis of educational policy and practice. This has created a growth in the use of randomised control trials (RCTs), which are considered an exemplary example of…

In multisite trials, learning about treatment effect variation across sites is critical for understanding where and for whom a program works. Unadjusted comparisons, however, capture "compositional" differences in the distributions of unit-level features as well as "contextual" differences in site-level features, including…

The concept of evidence, the demand for evidence-based practice and decision-making has for years dominated educational research. Concomitantly, the randomized controlled trials (RCT) that originally gained ground within the field of medicine have become the gold standard for empirical research and political reform within educational sciences, at…

In this commentary, we complement other constructive critiques of educational randomized control trials (RCTs) by calling attention to the commonly ignored role of context in causal mechanisms undergirding educational phenomena. We argue that evidence for the central role of context in causal mechanisms challenges the assumption that RCT findings…

In this commentary, we complement other constructive critiques of educational randomized control trials (RCTs) by calling attention to the commonly ignored role of context in causal mechanisms undergirding educational phenomena. We argue that evidence for the central role of context in causal mechanisms challenges the assumption that RCT findings…

When a researcher tests an educational program, product, or policy in a randomized controlled trial and detects a significant effect on an outcome, the intervention is usually classified as something that "works." When expected effects are not found, there is seldom an orderly and transparent analysis of plausible reasons why.…

Indirect comparisons are used to obtain estimates of relative effectiveness between two treatments that have not been compared in the same randomized controlled trial, but have instead been compared against a common comparator in separate trials. Standard indirect comparisons use only aggregate data, under the assumption that there are no…

This paper discusses, compares, and contrasts 4 different models for bringing evidence from randomised controlled trials (RCTs) into practice and into practical reasoning. I look at what questions the models can and cannot answer, what role they accord to RCT evidence, and what their possible attraction for practitioners might be. The models are…

Randomized controlled trials are a special case of designs using an unbiased instrument to take care of confounders even if they are unmeasured or unknown. Another example of studies using instrumental variables is the Mendelian experiment and Directed Acyclic Graphs show the power of such designs to enhance the internal validity. It is argued…

This paper represents a philosophical appraisal of Traditional Chinese Medicine (TCM) from the point of view of the philosophy of science. As it is generally the case with other versions of Traditional Medicine, rather than a coherent research program Traditional Chinese Medicine constitutes an array of various techniques and practices coupled…

In this article, Rachel Abenavoli, Natalia Rojas, Rebecca Unterman, Elise Cappella, Josh Wallack, and Pamela Morris argue that research-practice partnerships make it possible to rigorously study relevant policy questions in ways that would otherwise be infeasible. Randomized controlled trials of small-scale programs have shown us that early…

In this article, Rachel Abenavoli, Natalia Rojas, Rebecca Unterman, Elise Cappella, Josh Wallack, and Pamela Morris argue that research-practice partnerships make it possible to rigorously study relevant policy questions in ways that would otherwise be infeasible. Randomized controlled trials of small-scale programs have shown us that early…

Among econometricians, instrumental variable (IV) estimation is a commonly used technique to estimate the causal effect of a particular variable on a specified outcome. However, among applied researchers in the social sciences, IV estimation may not be well understood. Although there are several IV estimation primers from different fields, most…