Human raters' assessment of interpreting is a complex process. Previous researchers have mainly relied on verbal reports to examine this process. To advance our understanding, we conducted an empirical study, collecting raters' eye-movement and retrospection data in a computerised interpreting assessment in which three groups of raters (n = 35)…

The current study examines the contribution of racial congruence on young children's speech and language production to determine if African American children produce more or different language with an African American assessor, as opposed to a White assessor. Participants for the current study were selected from a larger group of children…

Behavior analysts typically rely on visual inspection of single-case experimental designs to make treatment decisions. However, visual inspection is subjective, which has led to the development of supplemental objective methods such as the conservative dual-criteria method. To replicate and extend a study conducted by Wolfe et al. (2018) on the…

The purpose of this study was to examine the effects of focus of attention instructions (i.e. focus on the solo, accompaniment, or collaboration) on listeners' performance evaluations. Participants (N = 159) were inservice band directors who listened to and evaluated the accuracy and expressivity of four excerpts of Haydn's "Concerto for…

The European Research Council (ERC) receives many high-quality applications, but funds only a few. We analyze how members of ERC review panels assess applications in the first, highly competitive step of evaluations for ERC Starting and Consolidator Grants. Drawing on interviews with ERC panel members in different fields, we show that they adopt a…

This article reflects on my personal experience as a TLI reviewer. It draws upon a decade of learning with and from colleagues, and connects the lessons learned from being both a reviewer and producer of SoTL output. I signpost the challenges and opportunities that belie the role of a TLI reviewer and celebrate the success the role brings. Through…

The purpose of this study was to explore how evaluators justify using story-based methodologies when examining causality. The two primary research questions of the study included: 1) what arguments are made by evaluators to justify the credibility of story-based causal methodologies to evaluation stakeholders; and 2) from the perspective of…

There is presently little consensus on how evaluators evaluate special education teachers. The lack of agreement may be due to the training of evaluators. Special education teachers work under various conditions with a diverse population. In this dissertation, we reviewed the challenges associated with evaluating special education teachers,…

For the first time in the history of the high stakes Leaving Certificate Established examination in Ireland, teachers graded and ranked their own students due to COVID-19 restrictions. In the wake of the process, a questionnaire and focus group interviews explored how teachers engaged with the Leaving Certificate Calculated Grades 2020 (CG2020)…

Practical constraints in rater-mediated assessments limit the availability of complete data. Instead, most scoring procedures include one or two ratings for each performance, with overlapping performances across raters or linking sets of multiple-choice items to facilitate model estimation. These incomplete scoring designs present challenges for…

Rater-mediated activities in educational research occur when an expert judge or rater utilizes an instrument to judge persons or items and generates scale scores. Scale scores are from a subjective judgment and must undergo a quality control measure called rating quality. Rating quality in this study is broadly defined as the extent to which…

Principles of skill acquisition dictate that raters should be provided with frequent feedback about their ratings. However, in current operational practice, raters rarely receive immediate feedback about their scores owing to the prohibitive effort required to generate such feedback. An approach for generating and administering feedback responses…

This introductory article describes how constructed response scoring is carried out, particularly the rater monitoring processes and illustrates three potential designs for conducting rater monitoring in an operational scoring project. The introduction also presents a framework for interpreting research conducted by those who study the constructed…

The emergence of ChatGPT as one of the most advanced chatbots and its ability to generate diverse data has given room for numerous discussions worldwide regarding its utility, particularly in advancing medical education and research. This study seeks to assess the performance of ChatGPT in medical biochemistry to evaluate its potential as an…

Recent times have seen the growth in the number and scope of interacting professional reform movements in science, centered on themes such as open research, research integrity, responsible research assessment, and responsible metrics. The responsible metrics movement identifies the growing influence of quantitative performance indicators as a…