Argumentation, a key scientific practice presented in the "Framework for K-12 Science Education," requires students to construct and critique arguments, but timely evaluation of arguments in large-scale classrooms is challenging. Recent work has shown the potential of automated scoring systems for open response assessments, leveraging…

The socioeconomic attainment gap grows as learners progress through the education system, meaning that it is at its widest when they reach the post-16 stage. In 2023, by the end of secondary school, socioeconomically disadvantaged pupils were 18.8 months behind their peers, whilst persistently disadvantaged pupils (those eligible for free school…

The term "meta-evaluation"--referring to the "evaluation of evaluations"--has been in the evaluation lexicon for a half-century. Despite this longevity, research on meta-evaluation is sparse and even more so for internal formative types of meta-evaluation. This exploratory study builds on our understanding of meta-evaluative…

This article reports research that examined the meaning of two broad evaluator competency domains. The first is "situational awareness" (SA) that focuses on understanding the unique contexts of evaluations and their users/stakeholders. The second is "interpersonal competence" (IC) that focuses on social skills needed for…

Purpose: The purpose of the self-study is two-fold: first, to aid in redressing the lack of attention given to the professional development i.e., the building of practical or work-related knowledge of examination invigilators and second, to forward the idea that engaging the examination invigilation process reflectively is an effective form of…

Practising internal auditors, including entry-level internal auditors, need face-to-face soft skills to effectively manage the increased complexity of their profession. Although many studies have highlighted the need for soft skills, none has identified the various categories of face-to-face soft skills required by entry-level internal auditors…

The Classroom Assessment Scoring System (CLASS; Pianta et al., 2008) is a popular measure of teacher-child interactions. Despite its prominence, CLASS scores have fairly weak relations with various child outcomes (e.g., Zaslow et al., 2010). One potential reason for these findings could be systematic differences in observer severity. As such, the…

The purpose of this study is to explore the use of unfolding models for evaluating the quality of ratings obtained in rater-mediated assessments. Two different judgmental processes can be used to conceptualize ratings: impersonal judgments and personal preferences. Impersonal judgments are typically expected in rater-mediated assessments, and…

Background: Multiple-choice item (MCI) assessments are burdensome for instructors to develop. Artificial intelligence (AI, e.g., ChatGPT) can streamline the process without sacrificing quality. The quality of AI-generated MCIs and human experts is comparable. However, whether the quality of AI-generated MCIs is equally good across various domain-…

This study investigates the validity and reliability of generative large language models (LLMs), specifically ChatGPT and Google's Bard, in grading student essays in higher education based on an analytical grading rubric. A total of 15 experienced English as a foreign language (EFL) instructors and two LLMs were asked to evaluate three student…

Broadly defined, academic language (AL) is a set of lexical-grammatical norms and registers commonly used in educational and academic discourse. Mastery of academic language in writing is an important aspect of writing instruction and assessment. The purpose of this study was to use Natural Language Processing (NLP) tools to examine the extent to…

There has been substantial research on doctoral supervision and examination, yet rarely a focus on what happens at the end-stage of the process when examiner feedback is received and addressed. This article reports survey findings (n = 262) from a study investigating supervisor perceptions about Australian end-stage doctoral examination processes.…

Purpose: The purpose of this paper is to study how quality assurance (QA) has impacted Argentina's higher education system, how QA tasks are reflected on the organizational structure of institutions, which kind of professional profiles the new QA staff assume and to what extent university life is reconfigured from these changes.…

In many performance assessments, one or two raters from the complete rater pool scores each performance, resulting in a sparse rating design, where there are limited observations of each rater relative to the complete sample of students. Although sparse rating designs can be constructed to facilitate estimation of student achievement, the…

In this Brief Report, we describe an evaluation of and revisions to a rubric adapted from the Jacobs et al.'s (1981) ESL COMPOSITION PROFILE, with four rubric categories and 20-point rating scales, in the context of an intensive English program writing placement test. Analysis of 4 years of rating data (2016-2021, including 434 essays) using…