Evaluators must attend to the destructive forces of racialization and racism to contribute to social transformation. Thus, evaluators are called to center culture, context, equity, and social justice during each step of the evaluation process. Here, we focus on the step(s) in which evaluators define program quality and specify evaluative lines of…

Task-based language assessment represents a major component of task-based language teaching syllabi. Current perspectives emphasise the importance of tasks in the assessment process, suggesting that adherence to influential models of language production during task design yields predictable test outcomes. The current study contends that the…

This scope-review presents the milestones of how Hierarchical Rater Models (HRMs) become operable to used in automated essay scoring (AES) to improve instructional evaluation. Although essay evaluations--a useful instrument for evaluating higher-order cognitive abilities--have always depended on human raters, concerns regarding rater bias,…

The rise of the digital era has led to a decline in handwriting as the primary mode of communication, resulting in negative effects on handwriting literacy, particularly in complex writing systems such as Chinese. The marginalization of handwriting has contributed to the deterioration of penmanship, defined as the ability to write aesthetically…

The effects of race can manifest in various ways in evaluation contexts, making it critical for evaluators to unpack how race and racism are "complex and destructive forces" for racially minoritized and Indigenous communities. The clarion calls by evaluators on the need for greater attention to issues of race and racism in evaluation…

Amid calls for evaluations to advance equity, there are ongoing debates, varied guidance, and limited empirical research on how evaluators practically attend to equity in their work. This article identifies ethical questions--about the right thing to do when there are multiple options--that arise when evaluators attend to equity and factors that…

Recent developments in the use of large-language models have led to substantial improvements in the accuracy of content-based automated scoring of free-text responses. The reported accuracy levels suggest that automated systems could have widespread applicability in assessment. However, before they are used in operational testing, other aspects of…

The availability of expert reviewers, essential for academic publishing, is increasingly under threat, due to workload pressures and lack of development pathways. This inquiry, undertaken by the editors of an emergent higher education journal, draws on reviewers' experiences as articulated in 'reviewer stories' and examines key questions around…

As part of the shift towards a more equitable research culture, funders are reconsidering traditional approaches to peer review. In doing so, they seek to minimize bias towards certain research ideas and researcher profiles, to ensure greater inclusion of disadvantaged groups, to improve review quality, to reduce burden, and to enable more…

Human ratings are ubiquitous in creativity research. Yet, the process of rating responses to creativity tasks -- typically several hundred or thousands of responses, per rater -- is often time-consuming and expensive. Planned missing data designs, where raters only rate a subset of the total number of responses, have been recently proposed as one…

With the generative artificial intelligence (AI) tool's remarkable capabilities in understanding and generating meaningful content, intriguing questions have been raised about its potential as an automated essay scoring (AES) system. One such tool is ChatGPT, which is capable of scoring any written work based on predefined criteria. However,…

In the PISA 2022 creative thinking test, students provide a response to a prompt, which is then coded by human raters as no credit, partial credit, or full credit. Like many large-scale educational testing frameworks, PISA uses the generalized partial credit model (GPCM) as a response model for these ordinal ratings. In this paper, we show that…

While significant attention has been given to test equating to ensure score comparability, limited research has explored equating methods for rater-mediated assessments, where human raters inherently introduce error. If not properly addressed, these errors can undermine score interchangeability and test validity. This study proposes an equating…

This study examines the feasibility and potential advantages of using large language models, in particular GPT-4o, to perform partial credit grading of large numbers of student written responses to introductory level physics problems. Students were instructed to write down verbal explanations of their reasoning process when solving one conceptual…

One-quarter of households in the state of Missouri are connected to an onsite sewage treatment system. These systems are necessary due to the risks human effluent poses to public health and our states' aquifers and waterways. For these systems to function correctly, a thorough and accurate evaluation of the soils and landforms onsite must be…