Randomized experiments are a strong design for establishing impact evidence because the random assignment mechanism theoretically allows confidence in attributing group differences to the intervention. Growth of randomized experiments within educational studies has been widely documented. However, randomized experiments within education have…

Formative evaluation provides stakeholders with timely feedback to support an intervention's improvement during implementation to maximize its effectiveness. We describe the qualitative methods that guided one study within a formative evaluation of a multicomponent care delivery intervention. We then describe the challenges and lessons learned…

Large-scale education interventions aimed at diminishing disparities and generating equitable learning outcomes are often complex, involving multiple components and intended impacts. Evaluating implementation of complex interventions is challenging because of the interactive and emergent nature of intervention components. Methods that build from…

Intervention scientists have used program theory-driven evaluation to design, implement, and assess the success of intervention programs for decades. However, interventions often are designed without the input of the community for which they are intended. The lack of incorporation of community members' voices that participate in various…

We assessed whether an adaptation of value-added analysis (VAA) can provide evidence on the relative effectiveness of interventions implemented in a large school district. We analyzed two datasets, one documenting interventions received by underperforming students, and one documenting interventions received by students in schools benefiting from…

Despite the increase in federal tiered evidence initiatives that require the use of rigorous evaluation designs, such as randomized experiments, there has been limited guidance in the evaluation literature on practical strategies to implement such studies successfully. This paper provides lessons learned in executing experiments in applied…

Evidence-based interventions hold promise for reducing gaps in health equity across diverse populations, but evidence about effectiveness within these populations lags behind the mainstream, often leaving opportunities to fulfill this promise unrealized. Mismatch between standard intervention outcomes research methods and the cultural and…

Many evaluations of programs tend to show few outcomes. One solution to this has been an increasing prominence of the movement that requires programs to implement evidence-based interventions (EBIs). But in a complex world with complex organizations and complex interventions, many challenges have arisen to the implementation of EBIs with fidelity…

Developmental evaluation (DE), essentially conceptualized by Patton over the past 30 years, is a promising evaluative approach intended to support social innovation and the deployment of complex interventions. Its use is often justified by the complex nature of the interventions being evaluated and the need to produce useful results in real time.…

The authors present a case study examining the potential for policies to be "evidence-based." To what extent is it possible to say that a decision to implement a complex social intervention is warranted on the basis of available empirical data? The case chosen is whether there is sufficient evidence to justify banning smoking in cars carrying…

In recent years, increased focus on the effectiveness and accountability of prevention and intervention programs has led to greater government funding for the implementation and spread of evidence-based health and human service delivery models. In particular, attention has been paid to programs that require significant infrastructure investment…

There is a growing recognition of the value of measuring fidelity of implementation (FOI) as a necessary part of evaluating interventions. However, evaluators do not have a shared conceptual understanding of what FOI is and how to measure it. Thus, the creation of FOI measures is typically a secondary focus and based on specific contexts and…

Multilevel modeling techniques facilitated examination of relationships between fidelity indicators and outcomes associated with a summer literacy intervention. Three-level growth models were specified to capture the extent to which students experienced instruction and to demonstrate the ways in which dosage-response relationships manifest in…

It has been acknowledged for several decades that programs interact with context. The nature of this interactivity, and how it defines a program, has not been adequately addressed. We view this lacuna as a function of the dominant theoretical perspectives guiding knowledge of program operations. We propose the actor-network theory (ANT) and its…

Program implementation data obtained from the repeated observation of teachers delivering one of two early childhood literacy programs to economically disadvantaged students in a large southwestern school district were analyzed to estimate protocol adherence levels at the onset of the intervention as well as the change in adherence over the…