Despite the rising popularity of big data, there is speculation that evaluators have been slow adopters of these new statistical approaches. Several possible reasons have been offered for why this is the case: ethical concerns, institutional capacity, and evaluator capacity and values. In this method note, we address one of these barriers and aim…

The purpose of this paper is to introduce the Five-Level Qualitative Data Analysis (5LQDA) method for ATLAS.ti as a way to intentionally design methodological approaches applicable to the field of evaluation. To demonstrate my analytical process using ATLAS.ti, I use examples from an existing evaluation of a STEM Peer Learning Assistant program.…

Several analytic strategies exist for opening up the "black box" to reveal more about what drives policy and program impacts. This article focuses on one of these strategies: the Analysis of Symmetrically-Predicted Endogenous Subgroups (ASPES). ASPES uses exogenous baseline data to identify endogenously-defined subgroups, keeping the…

Interest in the regression discontinuity (RD) design as an alternative to randomized control trials (RCTs) has grown in recent years. There is little practical guidance, however, on conditions that would lead to a successful RD evaluation or the utility of studies with underpowered RD designs. This article describes the use of RD design to…

This article presents discussion and recommendations on approaches to retrospectively evaluating development interventions in the long term through a systems lens. It is based on experiences from the implementation of an 18-month study to investigate the impact of development interventions on economic and social change over a 40-year period in the…

Qualitative data offer advantages to evaluators, including rich information about stakeholders' perspectives and experiences. However, qualitative data analysis is labor-intensive and slow, conflicting with evaluators' needs to provide punctual feedback to their clients. In this method note, we contribute to the literature on rapid evaluation and…

This study addresses validity issues in evaluation that stem from Ernest R. House's book, "Evaluating With Validity". The authors examine "American Journal of Evaluation" articles from 1980 to 2010 that report the results of policy and program evaluations. The authors classify these evaluations according to House's "major approaches" typology…

The continuously growing demand for program results has produced an increased need for evaluation capacity building (ECB). The "Integrative ECB Model" was developed to integrate concepts from existing ECB theory literature and to structure a synthesis of the empirical ECB literature. The study used a broad-based research synthesis method with…

In this article, we explore the evaluation capacity of today's nonprofit organizations. We report the findings of a cluster analysis that suggest that when it comes to evaluation, there are three types of nonprofit organizations. The first type of nonprofit organization is one that, by most accounts, is satisfied with their evaluation efforts.…

In attempting to use a realistic evaluation approach to explore the role of Community Parents in early parenting programs in Toronto, a novel technique was developed to analyze the links between contexts (C), mechanisms (M) and outcomes (O) directly from experienced practitioner interviews. Rather than coding the interviews into themes in terms of…

This article describes how job analysis, a method commonly used in personnel research and organizational psychology, provides a systematic method for documenting program staffing and service delivery that can improve evaluators' knowledge about program operations. Job analysis data can be used to increase evaluators' insight into how staffs…

Using the evaluation work on the W.K. Kellogg Foundation's Unleashing Resources Initiative as an example, in this article we explicate a general four-step model appropriate for multilevel evaluation alignment. We review the relevant literature, argue for the need for evaluation alignment in a multilevel context, explain the four-step model,…

A general inductive approach for analysis of qualitative evaluation data is described. The purposes for using an inductive approach are to (a) condense raw textual data into a brief, summary format; (b) establish clear links between the evaluation or research objectives and the summary findings derived from the raw data; and (c) develop a…

Suggests that performance measurement data already being collected could become more useful, at low additional cost, by further analyzing the relationships among the measures, especially across time. Using logic models and analyzing variability among program delivery units could provide evidence for hypothesized connections among delivery,…

The evaluation of an intervention delivered across multiple treatment sites presents a unique opportunity for evaluators to gauge the manner and degree to which the "impact" of treatment varies across implementation conditions and different target populations. However, the availability of implementation data for each treatment site, while…