Evaluators are often called upon to assess the effects of programs. To assess a program effect, evaluators need a clear understanding of how a program effect is defined. Arguably, the most widely used definition of a program effect is the counterfactual one. According to the counterfactual definition, a program effect is the difference between…

By design, randomized experiments (XPs) rule out bias from confounded selection of participants into conditions. Quasi-experiments (QEs) are often considered second-best because they do not share this benefit. However, when results from XPs are used to generalize causal impacts, the benefit from unconfounded selection into conditions may be offset…

Randomization of large number of participants to different treatment groups is often not a feasible or preferable way to answer questions of immediate interest to professional practice. Single case designs (SCDs) are a class of research designs that are experimental in nature but require only a few participants, all of whom receive the…

Since 2003, the Oral History Project Team has conducted interviews with individuals who have made particularly noteworthy contributions to the theory and practice of evaluation. In 2013, Mel Mark, Valerie Caracelli, and Miles McNall sat with Thomas Cook in Washington, D.C., during the American Evaluation Association (AEA) annual conference. The…

Treatment effect estimates from a "regression discontinuity design" (RDD) have high internal validity. However, the arguments that support the design apply to a subpopulation that is narrower and usually different from the population of substantive interest in evaluation research. The disconnect between RDD population and the…

Attention to evaluation quality is commonplace, even if sometimes implicit. Drawing on her 2010 Presidential Address to the American Evaluation Association, Leslie Cooksy suggests that evaluation quality depends, at least in part, on the intersection of three factors: (a) evaluator competency, (b) aspects of the evaluation environment or context,…

In the growing literature on the evaluation of program implementation, less has been said about evaluating program quality than about evaluating other aspects of program implementation. Furthermore, most articles and reports in the program-implementation evaluation literature have presented only brief descriptions of how implementation instruments…

Evaluators sometimes wish for a Fairy Godmother who would make decision makers pay attention to evaluation findings when choosing programs to implement. The U.S. Department of Education came close to creating such a Fairy Godmother when it required school districts to choose drug abuse prevention programs only if their effectiveness was supported…

Collaboration is a prerequisite for the sustainability of interagency programs, particularly those programs initially created with the support of time-limited grant-funding sources. From the perspective of evaluators, however, assessing collaboration among grant partners is often difficult. It is also challenging to present collaboration data to…

Shows how to bridge the gap between collaborative evaluations with extensive stakeholder participation and noncollaborative evaluations in which stakeholders do not participate to a great degree. Synthesizes research that shows that interaction with stakeholders helps enhance validity in noncollaborative evaluations. (SLD)

Fidelity may be defined as the extent to which delivery of an intervention adheres to the protocol or program model originally developed. Fidelity measurement has increasing significance for evaluation, treatment effectiveness research, and service administration. Yet few published studies using fidelity criteria provide details on the…

Data limitations severe enough to undermine the validity of findings discomfort evaluators whenever they occur. Unfortunately, such occurrences are not infrequent, faced not only by a "freshly minted graduate of a respected doctoral program in evaluation" but also by seasoned professionals (Bamberger, Rugh, & Mabry, forthcoming). Data may be…

Teaching difficulties often require creative approaches. Difficulties are often compounded when students in class show anxiety toward the material presented and, in particular, in the case of quantitative methods--"fear of numbers." This case presented itself in an advanced course in health behavior theory, where teaching the concepts of validity…