Given decision-makers often prioritize causal research that identifies the impact of treatments on the people they serve, a key question in education research is, "Does it work?". Today, however, researchers are paying increasing attention to successive questions that are equally important from a practical standpoint--not only does it…

Evaluators are often called upon to assess the effects of programs. To assess a program effect, evaluators need a clear understanding of how a program effect is defined. Arguably, the most widely used definition of a program effect is the counterfactual one. According to the counterfactual definition, a program effect is the difference between…

Researchers commonly interpret effect sizes by applying benchmarks proposed by Jacob Cohen over a half century ago. However, effects that are small by Cohen's standards are large relative to the impacts of most field-based interventions. These benchmarks also fail to consider important differences in study features, program costs, and scalability.…

Treatment effect estimates from a "regression discontinuity design" (RDD) have high internal validity. However, the arguments that support the design apply to a subpopulation that is narrower and usually different from the population of substantive interest in evaluation research. The disconnect between RDD population and the…

This article describes a modified importance-performance framework for use in evaluation of recreation-based experiential learning programs. Importance-performance analysis (IPA) provides an effective and readily applicable means of evaluating many programs, but the near universal satisfaction associated with recreation inhibits the use of IPA in…

Bullying is one of the most challenging issues facing students and schools worldwide. The disastrous consequences for victims and offenders are experienced daily by teachers and students and documented by numerous studies. The demand for evidence-based practice (EBP) in schools' anti-bullying work has increased in the last decade, consequently…

This report summarizes the complex research literature on quantitative methods for assessing how impacts of educational interventions on instructional practices and student learning differ across students, educators, and schools. It also provides technical guidance about the use and interpretation of these methods. The research topics addressed…

Community-based research (CBR) is an increasingly familiar approach to addressing social challenges. Nonetheless, the role it plays in attaining community impact is unclear and largely unstudied. Here the authors discuss an emerging framework aimed toward fostering community impact through university and community civic engagement. They describe…

Summarizes interesting aspects of the preceding chapters and the issues and concerns raised related to program theory evaluation. Suggests that evaluators may be fighting a losing battle in their efforts to shore up weak programs through the introduction of program theory evaluation, asserting that the development of strong program theory should…

Explores the relative strengths and weaknesses of program theory as a tool for inferring causality and outlines a five-stage approach that makes increased use of inductively built program theories as it takes more deliberate account of the varying levels of certainty that are required for evaluative conclusions. (Author/SLD)

Discusses the role that causal models play in program theory evaluation. Considers several types of causal relationships and discusses three causal models derived from systems theory, the theories of virtuous or vicious cycles, symptomatic solutions, and feedback delays. (SLD)

This article presents propensity score matching as a method to implement randomized conditions to analyze service effects using nonexperimental data. Most social work research is challenged to implement randomized clinical trials, whereas administrative and survey data are often available and can provide valuable information about services…

Program directors and evaluators need to address the important program accountability question of attribution of outcomes. This discussion is a beginning. Starting with some basics, such as the meaning of the program, approaches to program theory development, and the nature of attribution, the paper suggests three types of attribution. An…

Education researchers, practitioners, and policymakers alike are committed to identifying interventions that teach students more effectively. Increased emphasis on evaluation and accountability has increased desire for sound evaluations of these interventions; and at the same time, school-level data have become increasingly available. This article…

Statistical approaches to assess how prevention and intervention programs achieve their effects are described and illustrated through the evaluation of a health promotion program to reduce dietary cholesterol and a school-based drug prevention program. Analyses require the measurement of intervening or mediating variables to represent the…