In this essay, we describe the construction and use of the Cut-Score Operating Function in aiding standard setting decisions. The Cut-Score Operating Function shows the relation between the cut-score chosen and the consequent error rate. It allows error rates to be defined by multiple loss functions and will show the behavior of each loss…

This study discusses and presents an example of a use of spline functions to establish and report test scores using a moderated system of any number of cut scores. Our main goals include studying the need for and establishing moderated standards and creating a reporting scale that is referenced to all the standards. Our secondary goals are to make…

The Arizona English Language Learners Assessment (AZELLA) is used by the Arizona Department of Education to determine which children should receive English support services. AZELLA results are used to determine if children are either proficient in English or have English language skills in one of four pre-proficient categories (pre-emergent,…

Uses several thought experiments to explore the potential impact of the choice of criterion on the results of examinee-centered studies. Conclusions from these experiments are then used to examine the different cutting scores from several contrasting groups studies on the National Assessment of Educational Progress. (SLD)

Responds to a negative evaluation of the National Assessment of Educational Progress (NAEP) by the National Academy of Sciences (NAS) and asserts that a review of the evidence for the NAEP performance standards indicates that there is support for the current approach to NAEP standard setting. Considers the scholarship of the NAS evaluation…

Takes issue with support of the standard setting process of the National Assessment of Educational Progress (NAEP) advanced by critics of the evaluation of the standard setting process by the National Academy of Sciences and asserts that the achievement level descriptions developed for the NAEP no longer adhere to policy statements defining the…

Notes that the methods used to set standards for National Assessment of Education Progress (NAEP) tests suggest recommendations for state-level policymakers. Explains the national assessment, basic assumptions in setting performance standards on NAEP, selection of judges, standard-setting methodology for NAEP, and measurement issues in setting…

The process used to identify the provisional locations of points that would inform the National Assessment Governing Board in setting achievement levels for the science National Assessment of Educational Progress (NAEP) is described. This process includes identification of panelists for achievement-level setting and the method for converting…

Standard setting is a fairly widespread activity in educational and psychological measurement, but there is no formal psychometric theory to guide the development of standard setting methodology. This paper presents a conceptual framework for such a psychometric theory and uses the conceptual framework to analyze a number of methods for setting…

The Achievement-Levels Setting (ALS) process for the National Assessment of Educational Progress (NAEP) resulted in numerical cutscores on the NAEP score scale representing the performance standards for three achievement levels: Basic, Proficient, and Advanced. This paper focuses on an important, but less researched, aspect of the standard setting…

In 1999 a validation study of the 1998 Civics Assessment of the National Assessment of Educational Progress (NAEP) was conducted, combining a similarity classification study and a booklet classification study. The rationale was that if the teachers who had participated in the 5-day achievement levels-setting process could not use the descriptions…

Following a review of educational standard-setting experiences in the United States, the procedures used by the National Assessment of Educational Progress (NAEP) to arrive at a consensus on educational objectives are described. These objectives represent the only definitions of educational quality that are widely accepted by professional and lay…

Credentialing programs were surveyed to determine the procedures they use to set performance standards on multiple-choice and open-ended assessments. Implications of the various standard-setting approaches for the National Assessment of Educational Progress are discussed, and it is asserted that generalizing from standard-setting in professional…

In 1991 the National Assessment Governing Board (NAGB) released a report interpreting the achievement of U.S. students in mathematics on the 1990 National Assessment of Educational Progress in terms of a set of performance standards. The NAGB had been designing and implementing an approach to defining basic, proficient, and advanced levels of…

This paper looks at using descriptions of subject matter content to assist in the development and interpretation of student performance on the National Assessment of Educational Progress (NAEP). These descriptions of content, called achievement level descriptions (ALDs), were initially conceptualized as exemplary statements of the knowledge and…