In this article, we discuss the methodological implications of data and theory integration for Theory-Based Evaluation (TBE). TBE is a family of approaches to program evaluation that use program theories as instruments to answer questions about whether, how, and why a program works. Some of the groundwork about TBE has expressed the idea that a…

Program evaluation is an essential practice for providers of behavior analytic services, as it helps providers understand the extent to which they are achieving their intended mission to the community they serve. A proposed method for conducting such evaluations, is through the use of a consecutive case series design, for which cases are…

The purpose of the current research was to develop an electronic performance support system (EPSS) for quantitative data analysis by using design-based research. https://nicelanalizlericindesteksistemi.blogspot.com/ website was designed in Turkish to address the needs of educational researchers. There were four phases in the study. In the first…

A model that maps the requisite skills, or knowledge components, to the contents of an online course is necessary to implement many adaptive learning technologies. However, developing a skill model and tagging courseware contents with individual skills can be expensive and error prone. We propose a technology to automatically identify latent…

Due to repeated observations of an outcome behavior in N-of-1 or single-case design (SCD) intervention studies, the occurrence of missing scores is inevitable in such studies. Approximately 21% of SCD articles published in five reputable journals between 2015 and 2019 exhibited evidence of missing scores. Missing rates varied by designs, with the…

As is the case with many Centres for Teaching and Learning, prior to the global health crisis in 2020 our Centre primarily obtained feedback to inform its programming based on participation and satisfaction. As a result of the crisis, we could not rely on previous sources of information and needed to implement new strategies for evidencing the…

Purpose: The firms use training evaluation practices (TEPs) to determine the return of billions of dollars spent on employee training and development activities. The firms need to modernize the set of TEPs for evidence-based workforce management decisions. This study aims to examine a mediation mechanism to explain how human resource (HR)…

Analysis of symmetrically predicted endogenous subgroups (ASPES) is an approach to assessing heterogeneity in an ITT effect from a randomized experiment when an intermediate variable (one that is measured after random assignment and before outcomes) is hypothesized to be related to the ITT effect, but is only measured in one group. For example,…

We assessed whether an adaptation of value-added analysis (VAA) can provide evidence on the relative effectiveness of interventions implemented in a large school district. We analyzed two datasets, one documenting interventions received by underperforming students, and one documenting interventions received by students in schools benefiting from…

We assessed whether an adaptation of value-added analysis (VAA) can provide evidence on the relative effectiveness of interventions implemented in a large school district. We analyzed two datasets, one documenting interventions received by underperforming students, and one documenting interventions received by students in schools benefiting from…

Data-based individualization (DBI) is a process of collecting and analyzing data on students' response to intervention and then making intervention adaptations accordingly. Although this process can lead to better student outcomes, very few teachers are trained in the components of DBI, particularly in relation to behavior. Improving practice…

The purpose of this paper is to introduce the Five-Level Qualitative Data Analysis (5LQDA) method for ATLAS.ti as a way to intentionally design methodological approaches applicable to the field of evaluation. To demonstrate my analytical process using ATLAS.ti, I use examples from an existing evaluation of a STEM Peer Learning Assistant program.…

Data--done right--has the power to put schools on the path to true change. Rely on this research-backed resource to help you kickstart, implement, and sustain data-informed school-wide transformation. There are many ways to use student data, and the author's 10 steps offer practical, clear methods for establishing a data team, collecting relevant…

This guidance is for program and agency staff who have identified the need to develop a new data system or make major enhancements to an existing system. Key considerations are presented as questions to help guide staff through the initiation and planning process. The system initiation phase requires a broad team with the expertise and knowledge…

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some…