Background: For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require each document first be manually coded for constructs of interest by trained human raters. These hand-coded scores are then used as a measured outcome for an impact analysis, with the average scores of the treatment group…

The purpose of this study is to explore the views of autistic adults on randomised controlled trials, specifically on processes such as randomisation and blinding, to understand the barriers and facilitators for recruiting autistic people to randomised controlled trials involving medications. We conducted one-to-one interviews with 49 autistic…

The COVID-19 pandemic touched off an unprecedented search for vaccines and treatments. Without question, the development of vaccines to prevent COVID-19 was an enormous scientific accomplishment. Further, the RECOVERY and Solidarity trials identified effective treatments for COVID-19. But all was not success. The urgent need for COVID-19…

Although experimental evaluations have been labeled the "gold standard" of evidence for policy (U.S. Department of Education, 2003), evaluations without an analysis of costs are not sufficient for policymaking (Monk, 1995; Ross et al., 2007). Funding organizations now require cost-effectiveness data in most evaluations of effects. Yet,…

Traditionally, cluster randomized controlled trials are analyzed with the average intervention effect of interest. However, in populations that contain higher degrees of heterogeneity or variation may differ across different values of a covariate, which may not be optimal. Within education and social science contexts, exploring the variation in…

During the COVID-19 pandemic, the race to find an effective vaccine or treatment saw an 'extraordinary number' of clinical trials being conducted. While there were some key success stories, not all trials produced results that informed patient care. There was a significant amount of waste in clinical research during the pandemic which is said to…

By design, randomized experiments (XPs) rule out bias from confounded selection of participants into conditions. Quasi-experiments (QEs) are often considered second-best because they do not share this benefit. However, when results from XPs are used to generalize causal impacts, the benefit from unconfounded selection into conditions may be offset…

Sample size determination in multilevel randomized trials (MRTs) and multilevel regression discontinuity designs (MRDDs) can be complicated due to multilevel structure, monetary restrictions, differing marginal costs per treatment and control units, and range restrictions in sample size at one or more levels. These issues have sparked a set of…

In the complex world of educational policy, causal inferences will be debated. As we review non-experimental designs in educational policy, we focus on how to clarify and focus the terms of debate. We begin by presenting the potential outcomes/counterfactual framework and then describe approximations to the counterfactual generated from the…

Randomized A/B tests within online learning platforms represent an exciting direction in learning sciences. With minimal assumptions, they allow causal effect estimation without confounding bias and exact statistical inference even in small samples. However, often experimental samples and/or treatment effects are small, A/B tests are underpowered,…

Over the last twenty years, education researchers have increasingly conducted randomised experiments with the goal of informing the decisions of educators and policymakers. Such experiments have generally employed broad, consequential, standardised outcome measures in the hope that this would allow decisionmakers to compare effectiveness of…

Background: The COVID-19 pandemic and the subsequent public health response led to an unprecedented disruption in educational instruction in the U.S. and around the world. Many schools quickly moved to virtual learning for the bulk of the 2020 spring term and many states cancelled annual assessments of student learning. The 2020-21 school year…

Evaluating blocked randomized experiments from a potential outcomes perspective has two primary branches of work. The first focuses on larger blocks, with multiple treatment and control units in each block. The second focuses on matched pairs, with a single treatment and control unit in each block. These literatures not only provide different…

Randomized controlled trials are a special case of designs using an unbiased instrument to take care of confounders even if they are unmeasured or unknown. Another example of studies using instrumental variables is the Mendelian experiment and Directed Acyclic Graphs show the power of such designs to enhance the internal validity. It is argued…

Education decisionmakers need access to the best evidence about the effectiveness of education interventions, including practices, products, programs, and policies. It can be difficult, time consuming, and costly to access and draw conclusions from relevant studies about the effectiveness of interventions. The What Works Clearinghouse (WWC)…