This research evaluated the impact of a common modification to Angoff standard-setting exercises: the provision of examinee performance data. Data from 18 independent standard-setting panels across three different medical licensing examinations were examined to investigate whether and how the provision of performance information impacted judgments…

The purpose of the present study was to extend past work with the Angoff method for setting standards by examining judgments at the judge level rather than the panel level. The focus was on investigating the relationship between observed Angoff standard setting judgments and empirical conditional probabilities. This relationship has been used as a…

Despite being widely used and frequently studied, the Angoff standard setting procedure has received little attention with respect to an integral part of the process: how judges incorporate examinee performance data in the decision-making process. Without performance data, subject matter experts have considerable difficulty accurately making the…

One of the best-known methods for setting a benchmark standard on a test is that of Angoff and its modifications. When scored dichotomously, judges estimate the probability that a benchmark student has of answering each item correctly. As in most methods of standard setting, it is assumed implicitly that the unit of the latent scale of the…

This study investigated the extent to which the performance data format impacted data use in Angoff standard setting exercises. Judges from two standard settings (a total of five panels) were randomly assigned to one of two groups. The full-data group received two types of data: (1) the proportion of examinees selecting each option and (2) plots…

In order for standard setting to retain public confidence, it will be argued there are two important requirements. One is that the judges' allocation of students to performance bands would yield results broadly consistent with the expectation of the wider educational community. Secondly, in the absence of any change in educational performance,…

Absolute standard setting procedures are recommended for assessment in medical education. Absolute, test-centred standard setting procedures were introduced for written assessments in the Liverpool MBChB in 2001. The modified Angoff and Ebel methods have been used for short answer question-based and extended matching question-based papers,…

An analysis was made of the cognitive processes that support the judgments made in standard setting activities. These processes were conceived as having two components: forming the domain needed to pass the test and identifying the criterion level of performance to pass the test. In fact, these processes are interactive, and were separated for the…

A new approach for examining the quality of judgments from standard-setting judges using a Binomial Trials Model (BTM) is presented and illustrated with 26 judges from the Georgia High School Graduation Test. Results suggest that the BTM provides information not available from other methods. (SLD)

Judgmental standard setting methods, such as the W. H. Angoff (1971) method, use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore, the accuracy of these item performance estimates is crucial to the validity of the resulting MPS. Recent researchers, (L. A. Shephard 1994; J. Impara, 1997) have called…

School districts and credentialing agencies use information gathered in standard setting studies to establish minimum passing scores (MPS) for a variety of purposes. These scores may be used to make decisions ranging from subject remediation to licensure. Multiple standard setting methods may be used to provide a range of scores to the…

Judgmental standard setting approaches rely on the perceptions of experts about examinee performance on a test. Traditional standard setting methods ask panelists to estimate how well a randomly selected hypothetical examinee who is representative of a well-defined target group, usually a minimally competent candidate (MCC), will perform on each…

Two studies of variations of the Angoff method (W. Angoff, 1971) involving nine elementary school teachers in each case compared a yes/no estimation with a proportion correct estimation for setting cut scores. Both methods yielded essentially equal cut scores, but judges found the yes/no method easier to implement. (SLD)

Judgmental standard setting methods, such as the Angoff method (W. Angoff, 1971), use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore the accuracy of these item performance estimates is crucial to the validity of the resulting MPS. Recent researchers (L. Shepard, 1994; J. Impara, 1997) have called…

The relations between Angoff ratings (minimum passing levels) and the actual "p" values for borderline examinees were studied with 115 examinees taking the Certified Financial Planner examination. Findings do not suggest that the Angoff judges' task is nearly impossible, but they do suggest the need to improve standard-setting…