Scoring procedures for the constructed-response (CR) items in large-scale mixed-format educational assessments often involve checks for rater agreement or rater reliability. Although these analyses are important, researchers have documented rater effects that persist despite rater training and that are not always detected in rater agreement and…

The National Assessment Governing Board (NAGB) of the National Assessment of Educational Progress (NAEP) adopted mathematics achievement standards on May 11, 1991 that will be used to report results on the 1990 assessment of mathematics. NAEP achievement levels are descriptions, based on informed judgments, of what students should know and be able…

Across the field of educational assessment the case has been made for alternatives to the multiple-choice item type. Most of the alternative types of items require a subjective evaluation by a rater. The reliability of this subjective rating is a key component of these types of alternative items. In this paper, measures of reliability are…