This study aims to examine whether differences exist in the factors influencing the difficulty of scoring English summaries and determining scores based on the raters' attributes, and to collect candid opinions, considerations, and tentative suggestions for future improvements to the analytic rubric of summary writing for English learners. In this…

Assessing the content of learners' compositions is a common practice in second language (L2) writing assessment. However, the construct definition of content in L2 writing assessment potentially underrepresents the target competence in content and language integrated learning (CLIL), which aims to foster not only L2 proficiency but also critical…

As part of my work as an educator, I see the need to surface for discussion what might indeed be considered as acts of oppression on the part of peer reviewers when certain aspects of knowing and meaning are misrecognized, obscured, or suppressed. Drawing on observations concerning coercive and oppressive relational and educational practices found…

Argumentation, a key scientific practice presented in the "Framework for K-12 Science Education," requires students to construct and critique arguments, but timely evaluation of arguments in large-scale classrooms is challenging. Recent work has shown the potential of automated scoring systems for open response assessments, leveraging…

The purpose of this study is to explore the use of unfolding models for evaluating the quality of ratings obtained in rater-mediated assessments. Two different judgmental processes can be used to conceptualize ratings: impersonal judgments and personal preferences. Impersonal judgments are typically expected in rater-mediated assessments, and…

Practising internal auditors, including entry-level internal auditors, need face-to-face soft skills to effectively manage the increased complexity of their profession. Although many studies have highlighted the need for soft skills, none has identified the various categories of face-to-face soft skills required by entry-level internal auditors…

The Classroom Assessment Scoring System (CLASS; Pianta et al., 2008) is a popular measure of teacher-child interactions. Despite its prominence, CLASS scores have fairly weak relations with various child outcomes (e.g., Zaslow et al., 2010). One potential reason for these findings could be systematic differences in observer severity. As such, the…

There has been substantial research on doctoral supervision and examination, yet rarely a focus on what happens at the end-stage of the process when examiner feedback is received and addressed. This article reports survey findings (n = 262) from a study investigating supervisor perceptions about Australian end-stage doctoral examination processes.…

Purpose: The purpose of this paper is to study how quality assurance (QA) has impacted Argentina's higher education system, how QA tasks are reflected on the organizational structure of institutions, which kind of professional profiles the new QA staff assume and to what extent university life is reconfigured from these changes.…

In many performance assessments, one or two raters from the complete rater pool scores each performance, resulting in a sparse rating design, where there are limited observations of each rater relative to the complete sample of students. Although sparse rating designs can be constructed to facilitate estimation of student achievement, the…

This study investigates the validity and reliability of generative large language models (LLMs), specifically ChatGPT and Google's Bard, in grading student essays in higher education based on an analytical grading rubric. A total of 15 experienced English as a foreign language (EFL) instructors and two LLMs were asked to evaluate three student…

Broadly defined, academic language (AL) is a set of lexical-grammatical norms and registers commonly used in educational and academic discourse. Mastery of academic language in writing is an important aspect of writing instruction and assessment. The purpose of this study was to use Natural Language Processing (NLP) tools to examine the extent to…

Teacher performance scores inform education leaders' management of teacher human resources. However, prior research has implied that different interpretations of performance criteria between teachers and their evaluators suppress teacher development. Although research has examined teacher perceptions of performance scores and compared teacher…

The world has experienced rapid changes, leading to pressing issues such as environmental degradation, social inequality, and resource depletion. As a transdisciplinary field, evaluation has emerged as a crucial tool in addressing these challenges and promoting systemic change. However, concerns have been raised regarding the field's capacity to…

In this Brief Report, we describe an evaluation of and revisions to a rubric adapted from the Jacobs et al.'s (1981) ESL COMPOSITION PROFILE, with four rubric categories and 20-point rating scales, in the context of an intensive English program writing placement test. Analysis of 4 years of rating data (2016-2021, including 434 essays) using…