We explored three approaches to resolving or re-scoring constructed-response items in mixed-format assessments: rater agreement, person fit, and targeted double scoring (TDS). We used a simulation study to consider how the three approaches impact the psychometric properties of student achievement estimates, with an emphasis on person fit. We found…

Assessment is one of the five functions of the educational psychologist's (EP's) role, yet there is a dearth of research exploring its distinctive contribution to school-based practice, and a lack of definition about what it is. In this study, the assessment practices of EPs were compared with those of other educational professionals who had…

With the growing attention on large-scale educational testing and assessment, the ability to process substantial volumes of response data becomes crucial. Current estimation methods within item response theory (IRT), despite their high precision, often pose considerable computational burdens with large-scale data, leading to reduced computational…

The Mantel-Haenszel (MH) procedure is commonly used to detect items that function differentially for groups of examinees from various demographic and linguistic backgrounds--for example, in international assessments. As in some other DIF methods, the total score is used to match examinees on ability. In thin matching, each of the total score…

Differential item functioning (DIF) analysis is a key component in the evaluation of the fairness and validity of educational tests. The goal of this project was to review the status of ETS DIF analysis procedures, focusing on three aspects: (a) the nature and stringency of the statistical rules used to flag items, (b) the minimum sample size…

Most implementations of the Mantel-Haenszel differential item functioning procedure delete records with missing responses or replace missing responses with scores of 0. These treatments of missing data make strong assumptions about the causes of the missing data. Such assumptions may be particularly problematic when groups differ in their patterns…

How to compute and report subscores for a test that was originally designed for reporting scores on a unidimensional scale has been a topic of interest in recent years. In the research reported here, we describe an application of multidimensional item response theory to identify a subscore structure in a test designed for reporting results using a…

The global spread of national assessment testing activities, and the growing pressure to move beyond basic measures of participation in educational monitoring, means that student achievement measures are likely to become increasingly relevant indicators of systemic progress in the developing world. Using data from the CESSP project in Cambodia,…

Various applications of item response theory often require linking to achieve a common scale for item parameter estimates obtained from different groups. This article used a simulation to examine the relative performance of four different item response theory (IRT) linking procedures in a random groups equating design: concurrent calibration with…

The aim of the present research was to compare the use of multiple-choice questions (MCQs) as an examination method, to the examination based on constructed-response questions (CRQs). Despite that MCQs have an advantage concerning objectivity in the grading process and speed in production of results, they also introduce an error in the final…

This article demonstrates the use of a new class of model-free cumulative sum (CUSUM) statistics to detect person fit given the responses to a linear test. The fundamental statistic being accumulated is the likelihood ratio of two probabilities. The detection performance of this CUSUM scheme is compared to other model-free person-fit statistics…

This research investigates the effect of a web-based model, named "Practicing, Reflecting, and Revising with Web-based Assessment and Test Analysis system (P2R-WATA) Assessment Literacy Development Model," on enhancing assessment knowledge and perspectives of secondary in-service teachers, and adopts a single group experimental research…

The literature suggests that peer assessment contributes to the development of student learning and promotes ownership of assessment processes. These claims emerge from research conducted primarily in Western contexts. This exploratory paper reports on the perspectives that a class of Hong Kong primary school students and their teachers have on…

In four articles adapted from the Educational Testing Service (ETS) Research Report Series, Issue 2 of ETS Research Spotlight provides a small taste of the range of assessment-related research capabilities of the ETS Research and Development Division. Those articles cover assessment-related research aimed at developing models of student learning,…

Mathematical word problems represent a common item format for assessing student competencies. Automatic item generation (AIG) is an effective way of constructing many items with predictable difficulties, based on a set of predefined task parameters. The current study presents a framework for the automatic generation of probability word problems…