This article reviews the origins and use of the terms quasi-experiment and natural experiment. It demonstrates how the terms conflate whether variation in the independent variable of interest falls short of random with whether researchers find, rather than intervene to create, that variation. Using the lens of assignment--the process driving…

In cluster randomized evaluations, a treatment or intervention is randomly assigned to a set of clusters each with constituent individual units of observations (e.g., student units that attend schools, which are assigned to treatment). One consideration of these designs is how many units are needed per cluster to achieve adequate statistical…

In paired experiments, participants are grouped into pairs with similar characteristics, and one observation from each pair is randomly assigned to treatment. The resulting treatment and control groups should be well-balanced; however, there may still be small chance imbalances. Building on work for completely randomized experiments, we propose a…

Despite the increase in federal tiered evidence initiatives that require the use of rigorous evaluation designs, such as randomized experiments, there has been limited guidance in the evaluation literature on practical strategies to implement such studies successfully. This paper provides lessons learned in executing experiments in applied…

Incentivising has shown to improve participation in clinical trials. However, ethical concerns suggest that incentives may be coercive, obscure trial risks and encourage individuals to enrol in clinical trials for the wrong reasons. The aim of our study was to develop and pilot a discrete choice experiment (DCE) to explore and identify preferences…

This paper reviews the use of multiple trials, defined as multiple sites or multiple arms in a single evaluation and replications, in evaluating social programs. After defining key terms, the paper discusses the rationales for conducting multiple trials, which include increasing sample size to increase statistical power; identifying the most…

The past few years have seen a resurgence of faith in experimentation in education inquiry, and particularly in randomized controlled trials (RCTs). Proponents of such research have succeeded in bringing into common parlance the term "gold standard," which suggests that research emerging from any other design frame fails to achieve the…

Teacher coaching is a powerful form of professional learning that improves teaching practices and student achievement, yet little is known about the specific aspects of coaching programs that are more effective. Researchers used a blocked randomized experiment to study the effects of one-to-one coaching on teacher practice. When pooled across all…

This article discusses the nonuse, misuse, and proper use of pilot studies in experimental evaluation research. The authors first show that there is little theoretical, practical, or empirical guidance available to researchers who seek to incorporate pilot studies into experimental evaluation research designs. The authors then discuss how pilot…

Purpose: This multiyear experimental study was designed to examine (1) the causal impact of McREL International's Balanced Leadership® Professional Development (BLPD) program on school principals' learning, beliefs, and behaviors and (2) whether there were differences in the types of outcomes the professional development influenced. Outcomes…

The achievement gaps between poor and more affluent students are persistent and chronic, as many students living in poverty are also members of more isolated communities where dialects such as African American English and Southern Vernacular English are often spoken. Non-mainstream dialect use is associated with weaker literacy achievement. The…

Researchers are increasingly interested in working memory (WM) training. However, it is unclear whether it strengthens comprehension in young children who are at risk for learning difficulties. We conducted a modest study of whether the training of verbal WM would improve verbal WM and passage listening comprehension and whether training effects…

As online courses become more common, practitioners are in need of clear guidance on how to translate best educational practices into web-based instruction. Moreover, student engagement is a pressing concern in online courses, which often have high levels of dropout. Our goals in this work were to experimentally study routine instructional design…

We surveyed researchers in the health and exercise sciences to explore different areas and magnitudes of bias in researchers' decision making. Participants were presented with scenarios (testing a central hypothesis with p = 0.06 or p = 0.04) in a random order and surveyed about what they would do in each scenario. Participants showed significant…

Use of multivariate analysis (e.g., multivariate analysis of variance) is common when normally distributed outcomes are collected in intervention research. However, when mixed responses--a set of normal and binary outcomes--are collected, standard multivariate analyses are no longer suitable. While mixed responses are often obtained in…