We assessed whether an adaptation of value-added analysis (VAA) can provide evidence on the relative effectiveness of interventions implemented in a large school district. We analyzed two datasets, one documenting interventions received by underperforming students, and one documenting interventions received by students in schools benefiting from…

Tiered evidence initiatives are an important federal strategy to incentivize and accelerate the use of rigorous evidence in planning, implementing, and assessing social service investments. The Social Innovation Fund (SIF), a program of the Corporation for National and Community Service, adopted a public-private partnership approach to tiered…

Educational interventions are complex: Often they combine a diagnostic component (identifying student need) with a service component (ensuring appropriate educational resources are provided). This complexity raises challenges for program evaluation. These interventions, which we refer to as "service mediation interventions," affect…

Researchers have become increasingly interested in programs' main and interaction effects of two variables (A and B, e.g., two treatment variables or one treatment variable and one moderator) on outcomes. A challenge for estimating main and interaction effects is to eliminate selection bias across A-by-B groups. I introduce Rubin's causal model to…

There is a growing call for evaluation capacity building (ECB); although the area currently lacks a rich research base, there are few robust methods and practice through which to define it. The argument in this article is that the impact of evaluation is mediated by program stakeholders' engagement in evaluation activities. This mediation provides…

Although collaboration is recognized as an effective means to address multifaceted community issues, successful collaboration is difficult to achieve and failure is prevalent. To effectively collaborate, collaborators must recognize the strengths and weaknesses within their own efforts. Using Mattessich and colleagues' work as a springboard, a…

Using exogenous characteristics to identify endogenous subgroups, the approach discussed in this method note creates symmetric subsets within treatment and control groups, allowing the analysis to take advantage of an experimental design. In order to maintain treatment--control symmetry, however, prior work has posited that it is necessary to use…

Program evaluation is recognized as an essential skill set for practitioners in service-related fields, such as education, nonprofit management, social work, and public health. Recently, the need for a public workforce trained in evaluation has increased and is driven primarily by our nation's emphasis on accountability during a time when…

Evaluations of complex interventions such as sentencing guidelines provide an opportunity to understand the mechanisms by which policies and programs can impact intermediate and long-term outcomes. There is limited previous discussion of the underlying frameworks by which sentencing guidelines can impact outcomes such as crime rates. Guided by a…

This article presents the state of practice of evaluability assessment (EA) as represented in the published literature from 1986 to 2006. Twenty-three EA studies were located, showing that EA was conducted in a wide variety of programs, disciplines, and settings. Most studies employed document reviews, site visits, and interviews, common…

Cost studies are program evaluations that judge program worth by relating program costs to program benefits. There are three sets of strategies: cost-benefit, cost-effectiveness, and cost-utility analysis, although the last appears infrequently. The authors searched relevant databases to identify 103 cost studies in education and then reduced the…

Most K-12 evaluations are designed to make inferences about how a program implemented at the classroom or school level affects student learning outcomes and such inferences inherently involve hierarchical data structure. One methodological challenge for evaluators is linking program implementation factors typically measured at the classroom or…

Community-based organizations (CBOs), typically small and underfunded with transient staff members, are told by funders to care for clients and verify program value. To assist CBOs with evaluations that speak to program effectiveness, many funders wish to expand the evaluation capacity of CBO staff members so that evaluation will occur as long as…

Describes a three-step approach to teaching logic models that entails: (1) identification of antecedent conditions; (2) ensuring that program activities target antecedent conditions; and (3) a focus on measurement issues, depicting the indicators and objectives for outcomes included in the evaluation plan. (SLD)

The evaluation of an intervention delivered across multiple treatment sites presents a unique opportunity for evaluators to gauge the manner and degree to which the "impact" of treatment varies across implementation conditions and different target populations. However, the availability of implementation data for each treatment site, while…