Programs that improve teaching effectiveness represent a core strategy to improve student educational outcomes and close student achievement gaps. This article compiles empirical values of intraclass correlations for designing effective and efficient experimental studies evaluating the effects of these programs. The Early Childhood Longitudinal…

There is growing pressure to make efficacy experiments more useful. This requires attending to the twin goals of generalizing experimental results to those schools that will use the results and testing the intervention's theory of action. We show how electronic records, created naturally during the daily operation of technology-based…

Design-based methods have recently been developed as a way to analyze data from impact evaluations of interventions, programs, and policies. The impact estimators are derived using the building blocks of experimental designs with minimal assumptions, and have good statistical properties. The methods apply to randomized controlled trials (RCTs) and…

A new design-based theory has recently been developed to estimate impacts for randomized controlled trials (RCTs) and basic quasi-experimental designs (QEDs) for a wide range of designs used in social policy research (Imbens & Rubin, 2015; Schochet, 2016). These methods use the potential outcomes framework and known features of study designs…

Design-based methods have recently been developed as a way to analyze data from impact evaluations of interventions, programs, and policies (Imbens and Rubin, 2015; Schochet, 2015, 2016). The estimators are derived using the building blocks of experimental designs with minimal assumptions, and are unbiased and normally distributed in large samples…

In this article, we test an alternative approach to creating a counterfactual basis for estimating individual and average treatment effects. Instead of using control/comparison groups or before-measures, the so-called Counterfactual as Self-Estimated by Program Participants (CSEPP) relies on program participants' self-estimations of their own…

The purpose of the current work is to apply several main principles of the causal explanatory approach for establishing external validity to the experimental arena. By spanning the paradigm of the experimental approach and the school of program evaluation founded by Lee Cronbach and colleagues, the authors address the question of how research…

Randomized trials are a common way to provide rigorous evidence on the impacts of education programs. This article discusses the trade-offs associated with study designs that involve random assignment of students within schools and describes the experience from one such study of Teach for America (TFA). The TFA experiment faced challenges with…

Although random assignment is generally the preferred methodology in impact evaluations, it raises numerous ethical concerns, some of which are addressed by securing participants' informed consent. However, there has been little investigation of how consent is obtained in social experiments and the amount of information that can be conveyed--and…

It is argued that reports of evaluation activities often leave much to be desired because they fail to portray the real trial-and-error processes of the research study and do not report on the logic in use at the time of the inquiry. (SLD)

This paper examines the major methodological advantages of random assignment for the purpose of estimating the effectiveness of current manpower policy. It also reviews the claimed methodological and ethical objections to experiments. The authors argue that the offsetting gain from experimentation is the inherent reliability of experimental…

Proposes a methodology for analyzing the impacts of social programs on previously unexamined subgroups. The approach estimates the impact of programs on subgroups identified by a postreatment choice while maintaining the integrity of the experimental research design. (SLD)

Six potential problems with the use of randomized experiments to evaluate programs in the field are addressed. Problems include treatment dilution, treatment contamination or confounding, inaccurate case flow and power estimates, violations of the random assignment processes, changes in the environmental context, and changes in the treatment…

Asserts that experimental designs are the gold standard for research, addressing common criticisms of the application of experimental designs to education research, discussing why they have not been applied in education as frequently as they have been in other fields, examining circumstances under which experiments are appropriate, and offering…

This paper explores the impact that the evaluation industry has had on the development and implementation of social policy and programs, primarily as carried out by the U.S. Departments of Labor and Health and Human Services. In addition, major tools evaluators have developed and used, and the institutional arrangements through which they have…