As reporting of effect sizes in evaluation studies has proliferated, researchers and consumers of research need tools for interpreting or benchmarking the magnitude of those effect sizes that are relevant to the intervention, target population, and outcome measure being considered. Similarly, researchers planning education studies with social and…

It is a healthy exercise to debate the merits of using effect-size benchmarks to interpret research findings. However, these debates obscure a more central insight that emerges from empirical distributions of effect-size estimates in the literature: Efforts to improve education often fail to move the needle. I find that 36% of effect sizes from…

Purpose: This article provides benchmark data on within-group effect sizes from published randomized clinical trials supporting the efficacy of dialectical behavioral therapy (DBT) for borderline personality disorder (BPD) in adults aged 18-65 years. Method: Within-group effect sizes were calculated via the Glass approach and adjusted for sample…

This article provides benchmark data on within-group effect sizes from published randomized clinical trials that supported the efficacy of problem-solving therapy (PST) for depression among adults. Benchmarks are broken down by type of depression (major or minor), type of outcome measure (interview or self-report scale), whether PST was provided…

Cluster randomized experiments are now widely used to examine intervention effects in prevention science. It is meaningful to use empirical benchmarks for interpreting effect size in prevention science. The effect size (i.e., the standardized mean difference, calculated by the difference of the means between the treatment and control groups,…

Most evaluators have embraced the goal of evidence-based practice (EBP). Yet, many have criticized EBP review systems that prioritize randomized control trials and use various criteria to limit the studies examined. They suggest this could produce policy recommendations based on small, unrepresentative segments of the literature and recommend a…

Randomized experiments have long been established as the gold standard for addressing causal questions. However, experiments are not always feasible or desired, so observational methods are also needed. When multiple observations on the same variable are available, a repeated measures design may be used to assess whether a treatment administered…

The purpose of this study is to study the performance of different methods (inverse probability weighting and estimation of informative bounds) to control for differential attrition by comparing the results of different methods using two datasets: an original dataset from Portland Public Schools (PPS) subject to high rates of differential…

Researchers are increasingly using comparative interrupted time series (CITS) designs to estimate the effects of programs and policies when randomized controlled trials are not feasible. In a simple interrupted time series design, researchers compare the pre-treatment values of a treatment group time series to post-treatment values in order to…

Purpose: This article provides benchmark data on within-group effect sizes from published randomized clinical trials (RCTs) supporting the efficacy of trauma-focused cognitive behavioral therapy (TF-CBT) for traumatized children. Methods: Within-group effect-size benchmarks for symptoms of trauma, anxiety, and depression were calculated via the…

Policy-makers are frequently interested in understanding how effective a particular intervention may be for a specific (and often broad) population. In many fields, particularly education and social welfare, the ideal form of these evaluations is a large-scale randomized experiment. Recent research has highlighted that sites in these large-scale…

In 2013-14, the Wake County Public School System (WCPSS) launched Achieve3000 as a randomized controlled trial in 16 elementary schools. Achieve3000 is an early literacy program that differentiates non-fiction reading passages based on individual students' Lexile scores. Twoyear results show that Achieve3000 did not have a significant impact on…

The goal of this paper is to provide guidance for applied education researchers in using multi-level data to study the effects of interventions implemented at the school level. Two primary approaches are currently employed in observational studies of the effect of school-level interventions. One approach employs intact school matching: matching…

If children do not learn how to read in the first few years of primary school, they will struggle to complete the cycle and are at greater risk of dropping out. It is therefore crucial to identify and test interventions that have the potential of making a large impact, can be implemented quickly, and are affordable to be taken to scale by the…

The nested data structure inherent in education (i.e. students nested in schools nested in districts) makes intact school matching an appealing approach in observational studies of educational interventions and policies for both theoretical and practical purposes. This paper provides guidance to applied education researchers who are employing…