Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some…

This report from the Career and Technical Education (CTE) Research Network Lead team provides final results from an evaluability assessment of CTE programs that feasibly could be evaluated using a rigorous experimental design. Evaluability assessments (also called feasibility studies) are used in education and other fields, such as international…

Treatment effect estimates from a "regression discontinuity design" (RDD) have high internal validity. However, the arguments that support the design apply to a subpopulation that is narrower and usually different from the population of substantive interest in evaluation research. The disconnect between RDD population and the…

Card and Giuliano conducted a regression discontinuity study in a large Florida school district to investigate the magnitude of academic benefits of the district's gifted program. They found that for children identified as gifted through an intelligence test, the program provided few or no benefits. But children who were admitted to the gifted…

The effective evaluation of correctional programs is critically important. However, research in corrections rarely allows for the randomization of offenders to conditions of the study. This limitation compromises internal validity, and thus, causal conclusions can rarely be drawn. Increasingly, researchers are employing propensity score matching…

The report of the national response to intervention (RTI) evaluation study, conducted during 2011-2012, was released in November 2015. Anyone who has read the lengthy report can attest to its complexity and the design used in the study. Both these factors can influence the interpretation of the results from this evaluation. In this commentary, we…

This paper reviews literature on the rationale, challenges, and recommendations for choosing a nonequivalent comparison (NEC) group design when evaluating intervention effects. After reviewing frequently addressed threats to validity, the paper describes recommendations for strengthening the research design and how the recommendations were…

Background/Context: School finance litigation has often prompted funding reforms, but what happens as a result is the subject of considerable dispute. Purpose: This article explores design problems encountered in studies examining the nature and effects of those reforms. Analysis: After describing the development and current status of school…

This paper describes the analysis of the National Science Foundation (NSF) Math and Science Partnership (MSP) program's January 2008 Learning Network Conference (LNC). This study focuses on the methodologies used by the MSP community to generate evidence and seeks to understand topics of interest to the 320 LNC participants. The data set for this…

This article discusses the difficulties in conducting intervention research or evaluating intervention programs in a school setting. In particular, the problems associated with randomization and obtaining control groups are examined. The use of quasi-experimental designs, specifically a paired comparison design using the individual as his or her…

Evaluating school-based mental health services for children and youth with emotional disturbance (ED) has been a challenge for researchers. One particular challenge is the study design of using the student as the statistical unit of analysis, which in certain cases may lead to a violation of the "independence of error" assumption.…

Describes problems in implementing placebo controls in psychotherapy including the difficulty of ensuring that therapists and their patients will view both the experimental and placebo treatments as comparably credible. Considers six research issues stemming from the definitional requirement that the placebo control for the critical and specific…

This article discusses the use of propensity scoring in experimental program evaluations to estimate impacts for subgroups defined by program features and participants' program experiences. The authors discuss estimation issues and provide specification tests. They also discuss the use of an overlooked data collection design--obtaining predictions…

This note suggests a solution to the problem of achieving randomization in experimental settings where units deemed eligible for treatment "trickle in," that is, appear at any time. The solution permits replication of the experiment in order to test for time-dependent effects. (Author/CTM)

An examination of the scientific basis of both control group designs and comparative designs in outcome research reveals that comparative designs generally have fewer threats to validity and provide a more efficient means of control for nonspecific treatment factors suggesting that comparative studies warrant a larger role in the study of…