Setting performance standards on constructed-response assessments involving polytomously scored exercises presents a challenge for measurement practitioners. Some standard setting methods designed for use with multiple-choice, dichotomously scored assessments entail aggregating item performance estimates across a panel of experts. For these items,…

Four procedures were used to estimate a criterion-referenced standard for a multiple-choice examination developed by the National Board of Medical Examiners (NBME). Two experimental procedures, the NBME method and a modification of the Guerin method, and the Angoff and Ebel procedures were evaluated on the consistency of the estimates they…

The accuracy of standards obtained from judgmental methods is dependent on the quality of the judgments made by experts throughout the standard setting process. One important dimension of the quality of these judgments is the consistency of the judges' perceptions with item performance of minimally competent candidates. Several interrelated…

This special issue covers recent technical developments in the field of criterion-referenced testing. An introduction, six papers, and two commentaries dealing with test development, test score uses, and evaluation of scores review relevant literature, offer new models and/or results, and suggest directions for additional research. (SLD)

Minimum standards were established for the National Teacher Examinations (NTE) by teacher educators instructed in the use of the Angoff, Nedelsky, or Jaeger procedures. The anticipated failure rates, the psychometric characteristics of the ratings, and other factors suggest the Angoff procedure yields the most defensible standards for the NTE area…

This paper presents a comparison of two commonly used methods, Angoff (W. Angoff, 1971) and Bookmark (D. Lewis, H. Mitzel, and D. Green, 1996), for setting cut scores on selected response tests. This comparison is presented through an application to a grade 7 mathematics assessment in a suburban Midwestern school district. Training and operational…

Setting standards on educational tests is extremely challenging. The psychometric literature is replete with methods and guidelines for setting standards on educational tests; however, little attention has been paid to the process of setting standards on computerized adaptive tests (CATs). This lack of attention is unfortunate because CATs are…

This paper looks at using descriptions of subject matter content to assist in the development and interpretation of student performance on the National Assessment of Educational Progress (NAEP). These descriptions of content, called achievement level descriptions (ALDs), were initially conceptualized as exemplary statements of the knowledge and…

To address the issues associated with testing standards, sets of context-setting variables and technical matters associated with standard-setting are presented to assist groups or committees desiring to set standards in a systematic way. (Author/CM)

An ignored aspect of standard setting, namely the possibility that Angoff or Nedelsky judges specify inconsistent probabilities (e.g., low probabilities for easy items but large probabilities for hard items) is explored. A latent trait method is proposed to estimate such misspecifications, and an index of consistency is defined. (Author/PN)

The major forms of evidence that support a standard's credibility are reviewed, and what can be done over time and for different forms of an examination to enhance its comparability in a credentialing setting is outlined. Pass-fail decisions must be consistent to ensure a standard's credibility. (SLD)

Whether anchor item sets varying in difficulty and discrimination affect precision of cutting score equivalents generated through judge rescaling as much as equivalents from score equating was studied with 4 groups of experts and 250 and 1,000 examinees. Results indicate the robustness of judge rescaling and its superiority over equating. (SLD)

The comparability of Angoff-based item ratings on a general education test battery made by judges from within-content and across-content domains was studied. Results with 26 college faculty judges indicate that, at least for some tests, item ratings might be essentially equivalent regardless of judge's content specialty. (SLD)

A performance-standard setting procedure termed judgmental policy capturing (JPC) and its application are described. A study involving 12 panelists demonstrated the feasibility of the JPC method for setting performance standards for classroom teachers seeking certification from the National Board for Professional Teaching Standards. (SLD)

The three standard-setting approaches described in this special issue are summarized and contrasted: (1) judgmental policy capturing; (2) the extended Angoff method; and (3) the dominant profile method. An integrative summary of findings is followed by recommendations for modifying the methods. (SLD)