As is the case with many Centres for Teaching and Learning, prior to the global health crisis in 2020 our Centre primarily obtained feedback to inform its programming based on participation and satisfaction. As a result of the crisis, we could not rely on previous sources of information and needed to implement new strategies for evidencing the…

The purpose of this paper is to introduce the Five-Level Qualitative Data Analysis (5LQDA) method for ATLAS.ti as a way to intentionally design methodological approaches applicable to the field of evaluation. To demonstrate my analytical process using ATLAS.ti, I use examples from an existing evaluation of a STEM Peer Learning Assistant program.…

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some…

Youth apprenticeship is a strategy for building a more inclusive economy by creating affordable, reliable, and equitable pathways from high school to good jobs and college degrees. As youth apprenticeship programs develop and expand across the U.S., it is critical that practitioners develop plans for capturing, analyzing, and leveraging data at…

The dataset used in this report was created by combining Bridge to Finish intervention records, provided by the United Way of King County (UWKC), with student information provided by the State Board for Community and Technical Colleges (SBCTC). The intervention records span the terms from summer 2018 through fall 2021. SBCTC data include these…

This article examines the concept of credible evidence in Extension evaluations with specific attention to the measures and measurement strategies used to collect and create data. Credibility depends on multiple factors, including data quality and methodological rigor, characteristics of the stakeholder audience, stakeholder beliefs about the…

Short comparative interrupted times series (CITS) designs are increasingly being used in education research to assess the effectiveness of school-level interventions. These designs can be implemented relatively inexpensively, often drawing on publicly available data on aggregate school performance. However, the validity of this approach hinges on…

Expanded learning intermediaries are nonprofit organizations dedicated to making after-school and summer programs better and more accessible for children and youth. They do this by coordinating efforts and resources in a given community, knitting programs together into a cohesive system, helping individual programs function at a high level, and…

Ask Dr. Discovery is an NSF-funded study addressing the need for ongoing, large-scale museum evaluation while investigating new ways to encourage museum visitors to engage deeply with museum content. To realize these aims, we are developing and implementing a mobile app with two parts: (1) a front-end virtual scientist called Dr. Discovery (Dr. D)…

The "performance movement" has been a subject of enthusiasm and frustration for evaluators. Performance measurement, data analytics, and program evaluation have been treated as different tasks, and those addressing them speak their own languages in their own circles. We suggest that situating performance measurement and data analytics…

National College Access Network (NCAN) has heard repeatedly and consistently over the years that one of the best aspects of network membership is the ability to collaborate with and learn from other members. In an effort to further the transfer of ideas, in the summer of 2016 NCAN hosted a series of four Idea Incubators across the country, which…

The Local Child Outcomes Measurement System (L-COMS) is a framework that identifies seven key components of a high-quality child outcomes measurement system at the local level. The L-COMS framework was designed to apply to all local programs regardless of their specific child outcomes measurement approach. Development of the L-COMS originated with…

This short brief for education decision makers discusses three main factors that may contribute to a finding of no effects: failure of theory, failure of implementation, and failure of research design. It provides readers with questions to ask themselves to better understand "no effects" findings, and describes other contextual factors…

Analysis of student-level data to inform policy and promote student success is a core function of executive higher education agencies. Postsecondary data systems have expanded their collection of data elements for use by policymakers, institutional staff and the general public. State coordinating and governing boards use these data systems for…

Youth-led group work shifts power dynamics and repositions youth as leaders in driving the learning they envision for themselves. This shift calls into question how group facilitators measure outcomes of youth empowerment groups. Youth participatory action research (YPAR) has expanded the field of knowledge production by creating shared spaces…