Purpose: The firms use training evaluation practices (TEPs) to determine the return of billions of dollars spent on employee training and development activities. The firms need to modernize the set of TEPs for evidence-based workforce management decisions. This study aims to examine a mediation mechanism to explain how human resource (HR)…

The purpose of this paper is to introduce the Five-Level Qualitative Data Analysis (5LQDA) method for ATLAS.ti as a way to intentionally design methodological approaches applicable to the field of evaluation. To demonstrate my analytical process using ATLAS.ti, I use examples from an existing evaluation of a STEM Peer Learning Assistant program.…

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some…

Background: Mixed methods research designs in engineering education often frame the "mixed" aspect of the design from the perspective of data collection. However, intentional mixing throughout the research design--particularly during data analysis--may enable richer meta-inferences about a phenomenon. Purpose: This paper provides…

This article outlines how Master of Public Administration (MPA)/Master of Public Policy (MPP) programs can integrate a project-based learning opportunity to study curricular design and accreditation needs in their quantitative courses. Bridging together theory and the practical implications of data collection and analysis is important for…

These are the appendices for the report "The Effects of an Academic Language Program on Student Reading Outcomes." This study investigated WordGen Elementary, a program designed to improve fourth- and fifth-grade students' ability to understand and communicate academic language and their general reading skill. The program provider…

More aggregate data on school performance is available than ever before, opening up new possibilities for applied researchers interested in assessing the effectiveness of school-level interventions quickly and at a relatively low cost by implementing comparative interrupted times series (CITS) designs. We examine the extent to which effect…

Members of the Data Science Program at George Washington University (GWU) designed and implemented a tuition-free two-week summer camp at GWU for high-school students from the Washington Metro Area. The United States Department of Agriculture (USDA) Office of the Chief Information Officer and his staff were our main partners in the project. The…

Short comparative interrupted times series (CITS) designs are increasingly being used in education research to assess the effectiveness of school-level interventions. These designs can be implemented relatively inexpensively, often drawing on publicly available data on aggregate school performance. However, the validity of this approach hinges on…

Ask Dr. Discovery is an NSF-funded study addressing the need for ongoing, large-scale museum evaluation while investigating new ways to encourage museum visitors to engage deeply with museum content. To realize these aims, we are developing and implementing a mobile app with two parts: (1) a front-end virtual scientist called Dr. Discovery (Dr. D)…

Career Technology Education (CTE) program evaluations have been mostly completed for compliance and monitoring purposes. Hence, they have limited use in establishing program effectiveness or for program improvement. Moreover, engaging in program evaluations can be time consuming and costly, especially for programs that have not been evaluated in…

Interest in the regression discontinuity (RD) design as an alternative to randomized control trials (RCTs) has grown in recent years. There is little practical guidance, however, on conditions that would lead to a successful RD evaluation or the utility of studies with underpowered RD designs. This article describes the use of RD design to…

This short brief for education decision makers discusses three main factors that may contribute to a finding of no effects: failure of theory, failure of implementation, and failure of research design. It provides readers with questions to ask themselves to better understand "no effects" findings, and describes other contextual factors…

This timely book presents a data-driven approach to preventing and responding to school violence. As school violence receives increasing attention across the nation, the application of scientific knowledge is critical. For maximum effectiveness, transdisciplinary teams should use school data, logic models, and theories of change to design,…

The case study presented in this article is an example of an implementation evaluation. The evaluation investigated significant components of the implementation of a long-term environmental educational program. Direct observation, evaluation-specific survey data, and historical data were used to determine program integrity as identified by…