Randomized experiments are a strong design for establishing impact evidence because the random assignment mechanism theoretically allows confidence in attributing group differences to the intervention. Growth of randomized experiments within educational studies has been widely documented. However, randomized experiments within education have…

Evidence synthesis findings depend on the assumption that the included studies follow good clinical practice and results are not fabricated or false. Studies which are problematic due to scientific misconduct, poor research practice, or honest error may distort evidence synthesis findings. Authors of evidence synthesis need transparent mechanisms…

Single-case experimental designs (SCEDs) have a long history in clinical and educational disciplines. One underdeveloped area in advancing SCED design and analysis is understanding the process of how internal validity threats and operational concerns are avoided or mitigated. Two strategies to ameliorate such issues in SCED involve replication and…

Cluster randomized trials (CRTs) are commonly used to evaluate the causal effects of educational interventions, where the entire clusters (e.g., schools) are randomly assigned to treatment or control conditions. This study introduces statistical methods for designing and analyzing two-level (e.g., students nested within schools) and three-level…

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two…

Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to…

The COVID-19 pandemic touched off an unprecedented search for vaccines and treatments. Without question, the development of vaccines to prevent COVID-19 was an enormous scientific accomplishment. Further, the RECOVERY and Solidarity trials identified effective treatments for COVID-19. But all was not success. The urgent need for COVID-19…

During the COVID-19 pandemic, the race to find an effective vaccine or treatment saw an 'extraordinary number' of clinical trials being conducted. While there were some key success stories, not all trials produced results that informed patient care. There was a significant amount of waste in clinical research during the pandemic which is said to…

Evidence-based education aims to support policy makers choosing between potential interventions. This rarely involves considering each in isolation; instead, sets of evidence regarding many potential policy interventions are considered. Filtering a set on any quantity measured with error risks the "winner's curse": conditional on…

Background: Globally, conducting randomised controlled trials can be a complex endeavour. The complexity increases when including participants with cognitive or intellectual disabilities. A fuller understanding of the barriers and challenges that can be expected in such trials may help researchers to make their trials more inclusive for people…

Randomised trials are often funded by commercial companies and methodological studies support a widely held suspicion that commercial funding may influence trial results and conclusions. However, these studies often have a risk of confounding and reporting bias. The risk of confounding is markedly reduced in meta-epidemiological studies that…

A substantial number of randomized trials of educational interventions that have been conducted over the past two decades have produced null results, with either no impact or an unreliable estimate of impact on student achievement or other outcomes of interest. The investment of time and money spent implementing such trials warrants more useful…

Over the past 15 years, we have seen an increase in the use of cluster randomized trials (CRTs) to test the efficacy of educational interventions. These studies are often designed with the goal of determining whether a program works, or answering the what works question. Recently, the goals of these studies expanded to include for whom and under…

This chapter highlights the key assumptions underlying Randomized Control Trials (RCTs) and illustrates them with regard to the practice of RCTs in the realm of child and adolescent development. Given the prominence of RCTs in policy research, we analyze the possible ways in which these assumptions might not be met by single randomized…

In this response, we first show that Simpson's proposed analysis answers a different and less interesting question than ours. We then justify the choice of prior for our Bayes factors calculations, but we also demonstrate that the substantive conclusions of our article are not substantially affected by varying this choice.