Standard-setting studies utilizing procedures such as the Bookmark or Angoff methods are just one component of the complete standard-setting process. Decision makers ultimately must determine what they believe to be the most appropriate standard or cut score to use, employing the input of the standard-setting panelists as one piece of information…

Some writers in the measurement literature have been skeptical of the meaningfulness of achievement standards and described the standard-setting process as blatantly arbitrary. We argue that standard setting is more appropriately conceived of as a measurement process similar to student assessment. The construct being measured is the panelists'…

Judgmental standard setting methods have been criticized for the cognitive complexity of the judgment task that panelists are asked to complete. This study compared two methods designed to reduce this complexity: the yes/no method and the single-passage bookmark method. Two mock standard setting panel meetings were convened, one for each method,…

In their article "Innovations in Setting Performance Standards for K-12 Test-Based Accountability," Kristen Huff and Barbara S. Plake (2010) lay out three preconditions for continued investment in standard-setting methodology and practice, all focused on the sound development and use of achievement level descriptors (ALDs). Among these…

Standard setting methods such as the Angoff method rely on judgments of item characteristics; item response theory empirically estimates item characteristics and displays them in item characteristic curves (ICCs). This study evaluated several indexes of rater fit to ICCs as a method for judging rater accuracy in their estimates of expected item…

Standard setting is a method used to set cut scores on large-scale assessments. One of the most popular standard setting methods is the Bookmark method. In the Bookmark method, panelists are asked to envision a response probability (RP) criterion and move through a booklet of ordered items based on a RP criterion. This study investigates whether…

The aim of this study was to examine the effect of different standard setting methods on the size and composition of the borderline group, on the discrimination between different types of students and on the types of students passing with one method but failing with another. A total of 107 university students were classified into 4 different types…

Standard setting is a systematic process that uses a combination of judgmental and empirical procedures to make recommendations about where on the score continuum "cut scores" should be placed. Cut scores divide the score scale into categories consistent with the descriptions of student performance associated with multiple levels of achievement.…

Essential for the validity of the judgments in a standard-setting study is that they follow the implicit task assumptions. In the Angoff method, judgments are assumed to be inversely related to the difficulty of the items; contrasting-groups judgments are assumed to be positively related to the ability of the students. In the present study,…

The primary users of the current college ranking systems do not seem to be high-school students and families, but college presidents, board members, and development officers. As structured, the commercial ranking systems imply a precision that is not corroborated by research on what matters in college, nor can college quality be accurately summed…

Typical validation studies on standard setting models, most notably the Angoff and modified Angoff models, have ignored construct development, a critical aspect associated with all conceptualizations of measurement processes. Stone compared the Angoff and objective standard setting (OSS) models and found that Angoff failed to define a legitimate…

There is a strong quest in several countries including Australia for greater national consistency in education and intensifying interest in standards for reporting. Given this, it is important to make explicit the intended and unintended consequences of assessment reform strategies and the pressures to pervert and conform. In a policy context that…

This article describes four measurement tools that are of potential value for institutional researchers as greater demands are being placed upon their work. The author describes scale development, select methods for setting passing scores, validating passing scores and the topic of equating--both equipercentile and linear. Not only should…

The aim of this study is to standardize Autism Diagnostic Observation Schedule (ADOS) scores within a large sample to approximate an autism severity metric. Using a dataset of 1,415 individuals aged 2-16 years with autism spectrum disorders (ASD) or nonspectrum diagnoses, a subset of 1,807 assessments from 1,118 individuals with ASD were divided…

This paper describes how a state education system in Australia introduced standards-referenced assessments into its large-scale, high-stakes, curriculum-based examinations in a way that enables comparison of performance across time even though the examinations are different each year. It describes the multi-stage modified Angoff standard-setting…