The Campbellian validity typology has been used as a foundation for outcome evaluation and for developing evidence-based interventions for decades. As such, randomized control trials were preferred for outcome evaluation. However, some evaluators disagree with the validity typology's argument that randomized controlled trials as the best design…

During the COVID-19 pandemic, the race to find an effective vaccine or treatment saw an 'extraordinary number' of clinical trials being conducted. While there were some key success stories, not all trials produced results that informed patient care. There was a significant amount of waste in clinical research during the pandemic which is said to…

In this commentary, we complement other constructive critiques of educational randomized control trials (RCTs) by calling attention to the commonly ignored role of context in causal mechanisms undergirding educational phenomena. We argue that evidence for the central role of context in causal mechanisms challenges the assumption that RCT findings…

In this commentary, we complement other constructive critiques of educational randomized control trials (RCTs) by calling attention to the commonly ignored role of context in causal mechanisms undergirding educational phenomena. We argue that evidence for the central role of context in causal mechanisms challenges the assumption that RCT findings…

Across the evidence-based policy and practice (EBPP) community, including education, randomised controlled trials (RCTS) rank as the most "rigorous" evidence for causal conclusions. This paper argues that that is misleading. Only narrow conclusions about study populations can be warranted with the kind of "rigour" that RCTs…

Background/Context: When a researcher tests an educational program, product, or policy in a randomized controlled trial (RCT) and detects a significant effect on an outcome, the intervention is usually classified as something that "works." When the expected effects are not found, however, there is seldom an orderly and transparent…

Randomized trials play an important role in estimating the effect of a policy or social work program in a given population. While most trial designs benefit from strong internal validity, they often lack external validity, or generalizability, to the target population of interest. In other words, one can obtain an unbiased estimate of the study…

Background: The use of randomised controlled trials (RCTs) in education has increased significantly over the last 15 years. However, their use has also been subject to sustained and rather trenchant criticism from significant sections of the education research community. Key criticisms have included the claims that: it is not possible to undertake…

When cluster randomized experiments are analyzed as if units were independent, test statistics for treatment effects can be anticonservative. Hedges proposed a correction for such tests by scaling them to control their Type I error rate. This article generalizes the Hedges correction from a posttest-only experimental design to more common designs…

Improving academic, behavioural, and social outcomes of students through empirical research has been a firm commitment among researchers, policy-makers, and other professionals in education across Europe and the United States (U.S.). To assist in building scientific evidences, executive bodies such as the European Commission and the Institute for…

What Works Clearinghouse (WWC) quick reviews (QRs) are designed to provide education practitioners and policymakers with timely, preliminary objective assessments of the quality of the research evidence from recently released research papers and reports that have received coverage in the media. These reviews focus primarily on studies of the…

As evidence-based reform becomes increasingly important in educational policy, it is becoming essential to understand how research design might contribute to reported effect sizes in experiments evaluating educational programs. The purpose of this study was to examine how methodological features such as types of publication, sample sizes, and…

The consequences of failing core academic courses during the first year of high school are dire. More students fail courses in ninth grade than in any other grade, and a disproportionate number of these students subsequently drop out (Herlihy, 2007). As shown in Chicago and elsewhere, academic performance in core courses during the first year of…

Evidence-based practice, according to authoritative statements by the founders of this approach to health care, involves using the "best available" evidence in addition to clinical expertise and patient preferences to make decisions on the care of patients. However, many systematic reviewers interpret "best available" as…

Algebra is considered a key gatekeeper for higher-level mathematics course-taking in high school and for college enrollment (Adelman, 2006; Gamoran & Hannigan, 2000). Yet, algebra pass rates are consistently low in many places (Higgins, 2008; Ham & Walker, 1999; Helfand, 2006), including Chicago Public Schools (CPS). This is of particular…