Analysis of the differential treatment effects across targeted subgroups and contexts is a critical objective in many evaluations because it delineates for whom and under what conditions particular programs, therapies or treatments are effective. Unfortunately, it is unclear how to plan efficient and effective evaluations that include these…

Cluster randomized trials (CRTs), or studies in which intact groups of individuals are randomly assigned to a condition, are becoming more common in evaluation studies of educational programs. A specific type of CRT in which clusters are randomly assigned to treatment within blocks or sites, known as multisite cluster randomized trials (MSCRTs),…

Multilevel models are an increasingly popular method to analyze data that originate from a clustered or hierarchical structure. To effectively utilize multilevel models, one must have an adequately large number of clusters; otherwise, some model parameters will be estimated with bias. The goals for this paper are to (1) raise awareness of the…

Recent years have seen an increased interest in quantitative educational research studies that use random assignment (RA) to evaluate the causal impacts of educational interventions (Angrist, 2004). The multi-level structure of the public education system in the United States often leads to experimental designs where naturally occurring clusters…

For intervention studies involving binary treatment variables, procedures for power analysis have been worked out and computerized estimation tools are generally available. The purpose of this study is to: (1) develop the statistical formulations for calculating statistical power, minimum detectable effect size (MDES) and its confidence interval,…

Cluster randomized trials (CRTs), or studies in which intact groups of individuals are randomly assigned to a condition, are becoming more common in the evaluation of educational programs, policies, and practices. The website for the National Center for Education Evaluation and Regional Assistance (NCEE) reveals they have launched over 30…

Educational researchers commonly use the rule of thumb of "design effect smaller than 2" as the justification of not accounting for the multilevel or clustered structure in their data. The rule, however, has not yet been systematically studied in previous research. In the present study, we generated data from three different models…

In experimental designs with nested structures, entire groups (such as schools) are often assigned to treatment conditions. Key aspects of the design in these cluster-randomized experiments involve knowledge of the intraclass correlation structure, the effect size, and the sample sizes necessary to achieve adequate power to detect the treatment…

The analyses of J. E. Hunter (1983) were replicated with an expanded data set. The Hunter study, the basis of the Validity Generalization system used by the United States Employment Service, contained 515 General Aptitude Test Battery validation studies. The data set in this study included these and additional studies to bring the data set to 755…