Institutions of higher education are asked to consider how their work can advance equity in institutional outcomes. Assessment, too, has been asked to consider the ways in which traditional student assessment "privileges and validates certain types of learning and evidence of learning over others" (Montenegro and Jankowski 2017, p. 5).…

The purpose of this study was to explore how evaluators justify using story-based methodologies when examining causality. The two primary research questions of the study included: 1) what arguments are made by evaluators to justify the credibility of story-based causal methodologies to evaluation stakeholders; and 2) from the perspective of…

Practical constraints in rater-mediated assessments limit the availability of complete data. Instead, most scoring procedures include one or two ratings for each performance, with overlapping performances across raters or linking sets of multiple-choice items to facilitate model estimation. These incomplete scoring designs present challenges for…

Increasingly surveys are using interviewers to collect objective health measures, also known as biomeasures, to replace or supplement traditional self-reported health measures. However, the extent to which interviewers affect the (im)precision of biomeasurements is largely unknown. This article investigates interviewer effects on several…

The world has experienced rapid changes, leading to pressing issues such as environmental degradation, social inequality, and resource depletion. As a transdisciplinary field, evaluation has emerged as a crucial tool in addressing these challenges and promoting systemic change. However, concerns have been raised regarding the field's capacity to…

Rater effects, or raters' tendencies to assign ratings to performances that are different from the ratings that the performances warranted, are well documented in rater-mediated assessments across a variety of disciplines. In many real-data studies of rater effects, researchers have reported that raters exhibit more than one effect, such as a…

This guide will help researchers design and implement impact studies in education so that the findings are more generalizable to the study's target population. Guidance is provided on key steps that researchers can take, including defining the target population, selecting a sample of schools--and replacement schools, when needed--managing school…

Researchers have explored a variety of topics related to identifying and distinguishing among specific types of rater effects, as well as the implications of different types of incomplete data collection designs for rater-mediated assessments. In this study, we used simulated data to examine the sensitivity of latent trait model indicators of…

Implementation fidelity data indicate to what extent the delivered educational intervention (e.g., pedagogies, curricula) differs from the designed intervention (Gerstner & Finney, 2013; O'Donnell, 2008). Fidelity data help practitioners make more accurate inferences regarding program effectiveness (Dumas, Lynch, Laughlin, Smith, & Prinz,…

Evaluators can work with brief units of text-based data, such as open-ended survey responses, text messages, and social media postings. Online crowdsourcing is a promising method for quantifying large amounts of text-based data by engaging hundreds of people to categorize the data. To further develop and test this method, individuals were…

A curriculum map is a visualisation of relationships within and between a curriculum or curricula. Curriculum mapping refers to the method for creating and using the curriculum map, however this term is used broadly and encompasses a variety of methodological approaches. Often, researchers in the field of curriculum studies conduct curriculum…

An antecedent exercise (AE) intervention was conducted with two young children who demonstrated challenging behaviors during a large group circle time activity in a preschool classroom; outcomes were evaluated in the context of alternating treatments designs. Results from the initial visual analysis suggested no overlap between conditions, with…

Onscreen marking (OSM) has been used for the majority of Hong Kong public examinations since 2012. The current study compares marker reactions to OSM, ie, perceived ease of use and acceptance of OSM, against the backdrop of virtually all subject areas being marked on screen. The data were collected from three major sources: (1) survey data…

This article presents a content analysis of the 2013 School Leadership Program (SLP) grants. SLP projects provide a unique opportunity for participants in the field to explore innovative leadership preparation and development and their impact on program participants, schools, school districts, and students. The article begins with an overview of…

In today's world, much of what used to be individual work has become collaborative. Moreover, complex change initiatives often require individuals within and across organizations to team up to set and achieve meaningful goals. The role of researchers and evaluators is to offer support that can be used to strengthen the work of organizations and…