ERIC Number: EJ1463870
Record Type: Journal
Publication Date: 2025-Mar
Pages: 27
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0022-0655
EISSN: EISSN-1745-3984
Available Date: 2025-01-13
IRT Observed-Score Equating for Rater-Mediated Assessments Using a Hierarchical Rater Model
Tong Wu1,2; Stella Y. Kim2; Carl Westine2; Michelle Boyer3
Journal of Educational Measurement, v62 n1 p145-171 2025
While significant attention has been given to test equating to ensure score comparability, limited research has explored equating methods for rater-mediated assessments, where human raters inherently introduce error. If not properly addressed, these errors can undermine score interchangeability and test validity. This study proposes an equating method that accounts for rater errors by utilizing item response theory (IRT) observed-score equating with a hierarchical rater model (HRM). Its effectiveness is compared to an IRT observed-score equating method using the generalized partial credit model across 16 rater combinations with varying levels of rater bias and variability. The results indicate that equating performance depends on the interaction between rater bias and variability across forms. Both the proposed and traditional methods demonstrated robustness in terms of bias and RMSE when rater bias and variability were similar between forms, with a few exceptions. However, when rater errors varied significantly across forms, the proposed method consistently produced more stable equating results. Differences in standard error between the methods were minimal under most conditions.
Descriptors: Item Response Theory, Evaluators, Error of Measurement, Test Validity, Equated Scores, Models, Bias, Evaluation Methods, Robustness (Statistics), Hierarchical Linear Modeling
Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: 1Riverside Insights; 2University of North Carolina at Charlotte; 3Digital Recognition Corporation