Establishing cut scores using the Angoff method requires panelists to evaluate every item on a test and make a probability judgment. This can be time-consuming when there are large numbers of items on the test. Previous research using resampling studies suggest that it is possible to recommend stable Angoff-based cut score estimates using a…

Multiple matrix sampling is applied to a variation of Angoff's standard setting method. Thirty-six experts (internists) and 190 items were divided into five groups, and borderline examinee performance was estimated. There was some variability in the cutting scores produced by the individual groups, but various components were well estimated. (SLD)

Standards-based score reports interpret test performance with reference to cut scores defining categories like "below basic," "proficient," or "master." This article first develops a conceptual framework for validity arguments supporting such interpretations, then presents three applications. Two of these serve to introduce new standard-setting…

Setting standards for the purpose of certification is frequently performed using judgmental techniques such as the Angoff method. This study was performed to identify types of items that judges find hard to rate accurately, that is, types of items on which examinees perform differently than predicted by the judges. Once identified these item types…

To set a standard on the "beardedness" test (see TM 507 062) the probability that a student with a specific score will be judged as bearded must be estimated for each test score. To get an unbiased estimate of that probability, a representative sample of students at each test score level must be chosen. (BW)

Livingston's (TM 507 218) response to Rowley (TM 507 062) is compared with the original Zieky and Livingston formulation of the Contrasting Groups Method of setting standards. (BW)

A solution is offered to problems associated with the inequality in the manipulability of probabilities of classification errors of masters versus nonmasters, based on competency test results. Eschewing the typical arbitrary establishment of observed-score standards below 100 percent, the solution incorporates a self-correction of wrong answers.…

Alternative group judgment approaches to setting minimum competency standards were compared. Replication of results was possible for eight different tests (reading and mathematics, across four grade levels). The Kansas Competency Based Tests in reading and mathematics were administered statewide to students in grades two, four, six, and eight.…

Specific steps for applying the Angoff method are described. In the Angoff method, judges are asked to estimate the probabilities of minimally competent candidates' answering multiple choice test items correctly. Initial information must be obtained for designing the standard-setting process, beginning with the purpose of the examination and any…

Three studies assessed the psychometric characteristics of the Direct Standard Setting Method (DSSM). The Angoff technique was also used in each study. The DSSM requires judges to consider an examination 10 items at a time and determine the minimum items in that set a candidate should answer correctly to receive the credential. Nine judges set a…

An ignored aspect of standard setting, namely the possibility that Angoff or Nedelsky judges specify inconsistent probabilities (e.g., low probabilities for easy items but large probabilities for hard items) is explored. A latent trait method is proposed to estimate such misspecifications, and an index of consistency is defined. (Author/PN)

This paper explores common concerns about competency testing as they relate to the certification of high school graduates seeking a diploma in the United States. Competency testing is widespread in the United States, with 40 states engaged in competency testing in at least one grade. In general, and particularly for graduation requirements, the…

Standard setting practices in states using the NTE (formerly the National Teacher Examinations) were examined for 1987. The NTE is composed of two segments: (1) a Core Battery covering the communication skills and general knowledge skills; and (2) a test of professional knowledge about teaching. The processes used to establish passing scores for…

The sources of variability in the Angoff standard-setting procedure, when applied to the Alabama High School Graduation Examination (AHSGE), were examined. The sources of variability examined are judges, rounds (replications), competencies (items), and interactions among these three sources. After training, the judges were given a statement of a…

The implicit definition of competence and the inferential chain that links the standard-setting process to the decision outcomes of the method are considered for two classes of standard-setting procedures: those involving data-free judgments of items and those involving data-based judgment of items. The major underlying assumptions of competence…