Takes issue with support of the standard setting process of the National Assessment of Educational Progress (NAEP) advanced by critics of the evaluation of the standard setting process by the National Academy of Sciences and asserts that the achievement level descriptions developed for the NAEP no longer adhere to policy statements defining the…

Evaluated Angoff ratings from eight different occupational licensing examinations through generalizability theory to estimate the optimal number of raters. Results indicate that approximately 10 to 15 raters is an optimal target range. (SLD)

When one wants to set data reliability standards for a class of scientific inquiries or when one needs to compare and select among many different kinds of data with reliabilities that are crucial to a particular research undertaking, then one needs a single reliability coefficient that is adaptable to all or most situations. Work toward this goal…

Using meta-analysis, this research takes a look at studies included in a meta-analysis by R. Jaeger (1989) that compared the cut score set by one standard setting method with that set by another. This meta-analysis looks beyond Jaeger's studies to select 10 from the research literature. Each compared at least two types of standard setting method.…

Research studies that have investigated the use of mastery testing models and other related techniques are examined. A review of the procedures suggested for setting the pass fail point is also discussed. (Author/DEP)

Judgmental standard setting methods, such as the W. H. Angoff (1971) method, use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore, the accuracy of these item performance estimates is crucial to the validity of the resulting MPS. Recent researchers, (L. A. Shephard 1994; J. Impara, 1997) have called…

The kernel of the Angoff method of standard setting (W. Angoff, 1971) would seem to be the judgment of whether a minimally competent person could answer an item on a test correctly or not. So it would seem that any procedure that requires independent judgment of the correctness or incorrectness of a response to items for a minimally acceptable…

School districts and credentialing agencies use information gathered in standard setting studies to establish minimum passing scores (MPS) for a variety of purposes. These scores may be used to make decisions ranging from subject remediation to licensure. Multiple standard setting methods may be used to provide a range of scores to the…

Judgmental standard setting approaches rely on the perceptions of experts about examinee performance on a test. Traditional standard setting methods ask panelists to estimate how well a randomly selected hypothetical examinee who is representative of a well-defined target group, usually a minimally competent candidate (MCC), will perform on each…

Examined a method of student self- and peer assessment involving student construction of grading criteria using exemplars. Found that the use of exemplars can help students demonstrate greater understanding of both grading criteria and subject standards, and that the use of exemplars forms a focus for meaningful formative feedback. (EV)

Two studies of variations of the Angoff method (W. Angoff, 1971) involving nine elementary school teachers in each case compared a yes/no estimation with a proportion correct estimation for setting cut scores. Both methods yielded essentially equal cut scores, but judges found the yes/no method easier to implement. (SLD)

Examines criteria for choosing between test-centered and examinee-centered methods of standard setting in empirical terms and in terms of whether the method is consistent with the model of achievement underlying test design and interpretation and the assessment methods being used. Contains 35 references. (Author/SLD)

In cut score setting processes, subject matter experts are asked to make judgments about the likely performance of examinees at a targeted skill level. When cut scores are used in K-12 settings to separate students who have and have not mastered certain skills, the target examinee may be characterized as the barely proficient or barely master…

Standards-based score reports interpret test performance with reference to cut scores defining categories like "below basic," "proficient," or "master." This article first develops a conceptual framework for validity arguments supporting such interpretations, then presents three applications. Two of these serve to introduce new standard-setting…

Some of the different approaches to standard setting are discussed. Brief comments and references are offered concerning strategies that rely primarily on the use of expert judgment. Controversy surrounds methods that use expert judges, as well as those using test groups to set scores empirically. A minimax procedure developed by H. Huynh, an…