Intervention scientists have used program theory-driven evaluation to design, implement, and assess the success of intervention programs for decades. However, interventions often are designed without the input of the community for which they are intended. The lack of incorporation of community members' voices that participate in various…

This article presents an approach to use complex adaptive systems thinking to construct a theory of change to plan and evaluate transformational interventions. The article draws on the concepts of domains, scales, agents, and emergence to build a model of the system targeted by the intervention, to identify the chains of causality driving the…

We assessed whether an adaptation of value-added analysis (VAA) can provide evidence on the relative effectiveness of interventions implemented in a large school district. We analyzed two datasets, one documenting interventions received by underperforming students, and one documenting interventions received by students in schools benefiting from…

Despite the increase in federal tiered evidence initiatives that require the use of rigorous evaluation designs, such as randomized experiments, there has been limited guidance in the evaluation literature on practical strategies to implement such studies successfully. This paper provides lessons learned in executing experiments in applied…

Many evaluations of programs tend to show few outcomes. One solution to this has been an increasing prominence of the movement that requires programs to implement evidence-based interventions (EBIs). But in a complex world with complex organizations and complex interventions, many challenges have arisen to the implementation of EBIs with fidelity…

Educational interventions are complex: Often they combine a diagnostic component (identifying student need) with a service component (ensuring appropriate educational resources are provided). This complexity raises challenges for program evaluation. These interventions, which we refer to as "service mediation interventions," affect…

The need for evaluation capacity building (ECB) in military psychological health is apparent in light of the proliferation of newly developed, yet untested programs coupled with the lack of internal evaluation expertise. This study addresses these deficiencies by utilizing Preskill and Boyle's multidisciplinary ECB model within a post-traumatic…

Linking evaluation methods to the several phases of a program's life cycle can provide evaluation planners and funders with guidance about what types of evaluation are most appropriate over the trajectory of social and educational programs and other interventions. If methods are matched to the needs of program phases, evaluation can and should…

Prior evaluations of evaluation theory have been problematic due in part to confusion about what exactly is being assessed. After drawing distinctions among evaluation theories, models, and approaches, the author examines three aspects of evaluation models and considers how the evaluation of a model depends on which aspect is being examined. The…

Use of multivariate analysis (e.g., multivariate analysis of variance) is common when normally distributed outcomes are collected in intervention research. However, when mixed responses--a set of normal and binary outcomes--are collected, standard multivariate analyses are no longer suitable. While mixed responses are often obtained in…

Using exogenous characteristics to identify endogenous subgroups, the approach discussed in this method note creates symmetric subsets within treatment and control groups, allowing the analysis to take advantage of an experimental design. In order to maintain treatment--control symmetry, however, prior work has posited that it is necessary to use…

This article proposes an innovative approach to estimating the counterfactual without the necessity of generating information from either a control group or a before-measure. Building on the idea that program participants are capable of estimating the hypothetical state they would be in had they not participated, the basics of the Roy-Rubin model…

Longitudinal substance abuse research has often been compromised by high rates of attrition, thought to be the result of the lifestyle that often accompanies addiction. Several studies have used strategies including collection of locator information at the baseline assessment, verification of the information, and interim contacts prior to…

In recent years, increased focus on the effectiveness and accountability of prevention and intervention programs has led to greater government funding for the implementation and spread of evidence-based health and human service delivery models. In particular, attention has been paid to programs that require significant infrastructure investment…

This article introduces a new approach to program theory evaluation called theory-based stakeholder evaluation or the TSE model for short. Most theory-based approaches are program theory driven and some are stakeholder oriented as well. Practically, all of the latter fuse the program perceptions of the various stakeholder groups into one unitary…