Examined intra- and inter-rater consistency of item performance estimated from an Angoff standard setting over 2 years, with 29 panelists one year, and 30 the next. Results provide evidence that item performance estimates were consistent within and across panels within and across years. Factors that might have influenced this high degree of…

This paper reports the results of using several alternative methods of setting cut scores. The methods used were: (1) a variation of the Angoff method (1971); (2) a variation of the borderline group method; and (3) an advanced impact method (G. Dillon, 1996). The results discussed are from studies undertaken to set the cut scores for fourth grade…

Judgmental standard setting approaches rely on the perceptions of experts about examinee performance on a test. Traditional standard setting methods ask panelists to estimate how well a randomly selected hypothetical examinee who is representative of a well-defined target group, usually a minimally competent candidate (MCC), will perform on each…

Two studies of variations of the Angoff method (W. Angoff, 1971) involving nine elementary school teachers in each case compared a yes/no estimation with a proportion correct estimation for setting cut scores. Both methods yielded essentially equal cut scores, but judges found the yes/no method easier to implement. (SLD)

Judgmental standard setting methods, such as the Angoff method (W. Angoff, 1971), use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore the accuracy of these item performance estimates is crucial to the validity of the resulting MPS. Recent researchers (L. Shepard, 1994; J. Impara, 1997) have called…

A key component of the Angoff (W. Angoff, 1971) method for setting cut scores is the target examinee. Expert judges are asked, following training and discussion, to consider the ability and the likely performance of some subset of examinees and to then estimate, item by item, the likely performance of such examinees on the test for which a cut…

This study investigated the intrajudge consistency of Angoff-based item performance estimates. The examination used was a certification examination in an emergency medicine specialty. Ten expert panelists rated the same 24 items twice during an operational standard setting study. Results indicate that the panelists were highly consistent, in terms…

A study was conducted to explore empirically the effect of different definitions of the target examinee on the judgment of panelists setting a passing score. Two cut score studies were done in a school district for the same test within a 6-month period, and different definitions of the target candidate were provided for each study. In October…

Setting performance standards on constructed-response assessments involving polytomously scored exercises presents a challenge for measurement practitioners. Some standard setting methods designed for use with multiple-choice, dichotomously scored assessments entail aggregating item performance estimates across a panel of experts. For these items,…

This paper presents a comparison of two commonly used methods, Angoff (W. Angoff, 1971) and Bookmark (D. Lewis, H. Mitzel, and D. Green, 1996), for setting cut scores on selected response tests. This comparison is presented through an application to a grade 7 mathematics assessment in a suburban Midwestern school district. Training and operational…

Judgmental standard setting approaches rely on the perceptions of experts about examinee performance on a test. Traditional standard setting methods ask panelists to predict the probability that a randomly selected, hypothetical minimally competent candidate (MCC) will correctly answer test questions. Item performance predictions are difficult for…

Examined the reliability and accuracy of item performance estimates from an Angoff standard setting application with 29 panelists on 1 year and 30 in the next year. Results provide evidence that item performance estimates were both reasonable and reliable. Discusses factors that might have influenced the results. (SLD)

The American Association of School Administrators, the National Association of Elementary School Principals, the National Association of Secondary School Principals, and the National Council on Measurement in Education are cooperating on a project to develop competency standards for educational administrators in the area of student assessment. A…

This study investigated the cut score setting process as it occurred in two large Midwestern school districts, focusing on how the teachers who were the instruments by which cut scores were set experienced the process. Eight standard setting workshops using the Angoff approach were observed. Workshops for mathematics, reading, or writing at grades…

Many educational professionals have some knowledge of assessment practices, ranging from principles of test development and use to the practices associated with the use and interpretation of standardized and teacher-made tests. The skill levels associated with many important student assessment principles are, however, not consistent with the…