Institutions of higher education are asked to consider how their work can advance equity in institutional outcomes. Assessment, too, has been asked to consider the ways in which traditional student assessment "privileges and validates certain types of learning and evidence of learning over others" (Montenegro and Jankowski 2017, p. 5).…

Practical constraints in rater-mediated assessments limit the availability of complete data. Instead, most scoring procedures include one or two ratings for each performance, with overlapping performances across raters or linking sets of multiple-choice items to facilitate model estimation. These incomplete scoring designs present challenges for…

Increasingly surveys are using interviewers to collect objective health measures, also known as biomeasures, to replace or supplement traditional self-reported health measures. However, the extent to which interviewers affect the (im)precision of biomeasurements is largely unknown. This article investigates interviewer effects on several…

Rater effects, or raters' tendencies to assign ratings to performances that are different from the ratings that the performances warranted, are well documented in rater-mediated assessments across a variety of disciplines. In many real-data studies of rater effects, researchers have reported that raters exhibit more than one effect, such as a…

Researchers have explored a variety of topics related to identifying and distinguishing among specific types of rater effects, as well as the implications of different types of incomplete data collection designs for rater-mediated assessments. In this study, we used simulated data to examine the sensitivity of latent trait model indicators of…

Implementation fidelity data indicate to what extent the delivered educational intervention (e.g., pedagogies, curricula) differs from the designed intervention (Gerstner & Finney, 2013; O'Donnell, 2008). Fidelity data help practitioners make more accurate inferences regarding program effectiveness (Dumas, Lynch, Laughlin, Smith, & Prinz,…

Evaluators can work with brief units of text-based data, such as open-ended survey responses, text messages, and social media postings. Online crowdsourcing is a promising method for quantifying large amounts of text-based data by engaging hundreds of people to categorize the data. To further develop and test this method, individuals were…

An antecedent exercise (AE) intervention was conducted with two young children who demonstrated challenging behaviors during a large group circle time activity in a preschool classroom; outcomes were evaluated in the context of alternating treatments designs. Results from the initial visual analysis suggested no overlap between conditions, with…

Onscreen marking (OSM) has been used for the majority of Hong Kong public examinations since 2012. The current study compares marker reactions to OSM, ie, perceived ease of use and acceptance of OSM, against the backdrop of virtually all subject areas being marked on screen. The data were collected from three major sources: (1) survey data…

This article presents a content analysis of the 2013 School Leadership Program (SLP) grants. SLP projects provide a unique opportunity for participants in the field to explore innovative leadership preparation and development and their impact on program participants, schools, school districts, and students. The article begins with an overview of…

In today's world, much of what used to be individual work has become collaborative. Moreover, complex change initiatives often require individuals within and across organizations to team up to set and achieve meaningful goals. The role of researchers and evaluators is to offer support that can be used to strengthen the work of organizations and…

Randomized controlled trials are the gold standard in research but may not fully explain or predict outcome variations in community-based interventions. Demonstrating efficacy of externally driven programs in well-controlled environments may not translate to community-based implementation where resources and priorities vary. A bottom-up evaluation…

Program and/or course evaluation is a process in which different types of data are collected systematically in order to study the virtues and weaknesses of a language instruction program. Program evaluation is, in fact, one of the essential aspects of any curriculum. It is a kind of quality control in which various aspects of an instructional…

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff,…

This study is a content analysis of selected federally funded leadership preparation program evaluation proposals for the 2008, 2009, and 2010 School Leadership Program grants. The United States Department of Education (USDE) began awarding funding in 2002 to develop and implement preparation programs for aspiring and current assistant principals…