There is a burgeoning body of evidence on the average impacts of educational programs. Yet, for many local decision makers, because impacts can vary across sites, the question of whether a certain program will work in their particular district or school remains. This article addresses the question of the generalizability of large-scale average…

Over the last few decades, there has been an increase in the number of large federally funded transdisciplinary programs and initiatives. Scholars have identified a need to develop frameworks, methodologies, and tools to evaluate the effectiveness of these large collaborative initiatives, providing precise ways to understand and assess the…

The purpose of this paper is to introduce the Five-Level Qualitative Data Analysis (5LQDA) method for ATLAS.ti as a way to intentionally design methodological approaches applicable to the field of evaluation. To demonstrate my analytical process using ATLAS.ti, I use examples from an existing evaluation of a STEM Peer Learning Assistant program.…

The term "meta-evaluation"--referring to the "evaluation of evaluations"--has been in the evaluation lexicon for a half-century. Despite this longevity, research on meta-evaluation is sparse and even more so for internal formative types of meta-evaluation. This exploratory study builds on our understanding of meta-evaluative…

Although experts agree that diverse stakeholder inclusion enhances quality and equity in evaluation designs and implementation, diverse voices are often omitted. Particularly antithetical to principles of youth character development, evaluations for these programs should strive to include voices from various social, economic, community, and…

Needs assessments (NAs) for marginalized communities would ideally contextualize needs in the sociocultural context, use agency-supportive methods, and result in liberatory action planning. This article develops the Transformative Needs Assessment With Marginalized Communities (TNAMC) using a mixed-methods approach that examines internal and…

Advocacy outcomes are not self-evident. Identifying advocacy outcomes is extremely difficult because they are often intangible, arising from (personal) interactions, and they are not always traceable. This challenges conventional evaluation methods, which is recognized in the advocacy evaluation literature. However, current evaluation methods…

Evaluators are becoming increasingly aware that, to provide maximum benefit to the programs they evaluate, they must address the systems within which the programs work. However, evaluation practice is often influenced by beliefs and assumptions that are rooted in an understanding of these systems as stable and predictable. We identify four traps…

The notions of indigenous knowledge and cultural philosophies are becoming ubiquitous in many social inquiries, and evaluation is no exception. Nonetheless, the pursuit to embed relevant indigenous philosophies in contemporary evaluation has yet to succeed. In this article, we discuss indigenous relational philosophies, approaches, and practices…

Program evaluations that lack experimental design often fail to produce evidence of impact because there is no available control group. Theory-based evaluations can generate evidence of a program's causal effects if evaluators collect evidence along the theorized causal chain and identify possible competing causes. However, few methods are…

This article examines factors that could have influenced whether evaluations of U.S. government--funded foreign assistance programs completed in 2015 had considered unintended consequences. Logit regression models indicate that the odds of considering unintended consequences were increased when all or most of seven standard data collection methods…

Intervention programs are open systems that must withstand the disturbances of the environments in which they are implemented if they are to thrive. Stakeholders can benefit from proactive insights related to potential program malfunction or failure. Evaluators have not developed approaches to serve this preventive function. This study proposes…

The adversary evaluation model emerged in a context that favored democratic debate. It was successfully used in a variety of sectors following its inception, but it was abandoned once neo-liberal thinking and goal achievement approaches became dominant. It is time to give it a second chance. The judicial evaluation model (JEM) relies on human…

Transformative change is needed if the world is to achieve the sustainable development goals. Such change requires attention to culture, ethics, and values. We discuss the need to be responsive to the voices of Indigenous scholars in terms of the assumptions that guide methodological choices in the evaluation of international initiatives. We…

Community-based partnerships are integral to mental health programming and research. However, there are limited published guidelines that apply the principles of community-based participatory research (CBPR), especially within the context of supporting vulnerable youth populations. This article demonstrates the application of the CBPR principles…