This study considers the estimation of marginal reliability and conditional accuracy measures using a generalized recursion procedure with several IRT-based ability and score estimators. The estimators include MLE, TCC, and EAP abilities, and corresponding test scores obtained with different weightings of the item scores. We consider reliability…

Using item-response theory to model rater effects provides an alternative solution for rater monitoring and diagnosis, compared to using standard performance metrics. In order to fit such models, the ratings data must be sufficiently connected in order to estimate rater effects. Due to popular rating designs used in large-scale testing scenarios,…

We introduce a short answer scoring engine made up of an ensemble of deep neural networks and a Latent Semantic Analysis-based model to score short constructed responses for a large suite of questions from a national assessment program. We evaluate the performance of the engine and show that the engine achieves above-human-level performance on a…

Purpose: The authors constructed an automatic essay scoring (AES) model in a discussion forum where the result was compared with scores given by human evaluators. This research proposes essay scoring, which is conducted through two parameters, semantic and keyword similarities, using a SentenceTransformers pre-trained model that can construct the…

This study aims to examine whether differences exist in the factors influencing the difficulty of scoring English summaries and determining scores based on the raters' attributes, and to collect candid opinions, considerations, and tentative suggestions for future improvements to the analytic rubric of summary writing for English learners. In this…

Machine learning algorithms that automatically score scientific explanations can be used to measure students' conceptual understanding, identify gaps in their reasoning, and provide them with timely and individualized feedback. This paper presents the results of a study that uses Hebrew NLP to automatically score student explanations in Biology…

The formative use of rubrics seems to have the potential of promoting student learning, supposedly by making expectations and criteria explicit. However, there is a variation in effects on how well students perform on academic tasks when supported by rubrics. The aim of this study was to identify factors in rubric interventions that may…

Assessments in higher education healthcare programmes can be challenging because they not only need to be fair, valid, and transparent, but it is also necessary to gauge safety, practical skill competence, and professionalism. One way to help maximise validity in practical assessments is to utilise 'standard setting' which aims to set a fair 'cut…

Advanced practices for summative exam development and post-exam analysis are proven to be effective but aren't always practical, and, even when these are applied to some degree, exams remain inherently imperfect measures of student ability. Instructors may thus deem it necessary to adjust overall exam scores to account for aspects of an exam that…

Building on previous research, the authors of this article propose the use of a series of interrelated rubrics for student assessment and self-assessment in the dance technique class. These rubrics are based on learning strategies the authors observed students demonstrate. The authors propose that bringing students' attention to these strategies…

Rubrics (e.g., Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics) that measure specific skills exist, and researchers have demonstrated their benefits; however, most of them were designed for use with undergraduate students. Although some rubrics have been created to assess dissertations and oral defenses, few have been…

This scope-review presents the milestones of how Hierarchical Rater Models (HRMs) become operable to used in automated essay scoring (AES) to improve instructional evaluation. Although essay evaluations--a useful instrument for evaluating higher-order cognitive abilities--have always depended on human raters, concerns regarding rater bias,…

ACT announced a series of enhancements designed to modernize the ACT test and offer students more choice and flexibility in demonstrating their readiness for life after high school. The enhancements provide students more flexibility by allowing them to choose whether to take the science assessment, thereby reducing the test length by up to…

Evaluating the performance of Large Language Models (LLMs) is a critical yet challenging task, particularly when aiming to avoid subjective assessments. This paper proposes a framework for leveraging subjective metrics derived from the class textual materials across different semesters to assess LLM outputs across various tasks. By utilizing…

Background: STEAM education emphasizes transdisciplinary application and practice; however, there is a lack of comprehensive assessment tools for evaluating K-12 students' transdisciplinary learning. Given the cross-domain instructional and learning approaches in STEAM education, this study aims to develop and validate the Learn, Use, Practice,…