This article examines three typical approaches to alternate assessment for students with significant cognitive disabilities--portfolios, performance assessments, and rating scales. A detailed analysis of common and unique design features of these approaches is provided, including features of each approach that influence the psychometric quality of…

Many approaches to standard-setting use item calibration and student score estimation results to structure panelists' tasks. However, this requires collecting standard-setting judgments after the item analysis results are available. The Scoring Guide Alignment approach collects standard-setting judgments during the scoring sessions from teachers…

Psychometricians continue to introduce new approaches to setting cut scores for educational assessments in an attempt to improve on current methods. In this paper we describe the Item-Descriptor (ID) Matching method, a method based on IRT item mapping. In ID Matching, test content area experts match items (i.e., their judgments about the knowledge…

In an Angoff standard setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item in the test. In many cases, these item performance estimates are made twice, with information shared with the panelists between estimates. Especially for long tests, this…

In an Angoff standard-setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item constituting the test. In many cases, these item performance estimates are made twice, with information shared with the judges between estimates. Especially for long tests,…

This article reviews the empirical literature on 9 topics about the modified Angoff standard-setting method that have been studied repeatedly in the literature, while taking into consideration the methodological warrant for the findings on the topics. It concludes that we can be reasonably confident about selecting the appropriate number of judges…

The authors used a 16-item Ethical Intent Checklist, developed from the American Counseling Association's (1999) Ethical Standards for Internet Online Counseling, to assess the current practices of 88 online counseling Web sites. Results showed fewer than half of online counselors were following the accepted practice on 8 of the 16 items. Online…

In the Angoff standard setting procedure, subject matter experts (SMEs) estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item comprising the test. In many cases, these item performance estimates are made twice, with information shared with SMEs between estimates. This…

Uses several thought experiments to explore the potential impact of the choice of criterion on the results of examinee-centered studies. Conclusions from these experiments are then used to examine the different cutting scores from several contrasting groups studies on the National Assessment of Educational Progress. (SLD)

The authors provide a reference on how to set standards on tests in education, licensure, and certification. This book is comprehensive in scope, practical in nature, and definitive in terms of cataloguing the essential conceptual and procedural fundamentals of setting performance standards. It: (1) Offers a detailed description of procedures; (2)…

Judge-mediated examinations are defined as those for which expert evaluation (using rubrics) is required to determine correctness, completeness, and reasonability of test-taker responses. The use of multifaceted Rasch modeling has led to improvements in the reliability of scoring such examinations. The establishment of criterion-referenced…

This issue brief outlines the immediate need for more accurate dropout and graduation data, while providing a snapshot of work currently underway. By drawing on two prominent methods for calculating graduation rates: the National Governors Association's endorsed longitudinal approach and the Averaged Freshman Graduation Rate (AFGR), this brief…

In the traditional Angoff Standard Setting Method, experts are instructed to predict the possibility that a randomly selected, hypothetical minimally competent candidate will be able to answer each multiple choice question in the test correctly. These item performance estimates are averaged across panelists and aggregated to determine the minimum…

Standard-setting research has yielded a rich array of more than 50 standard-setting procedures, but practitioners are likely to be confused about which to use. By synthesizing the accumulated research on standard setting and progress monitoring, this study developed a three-dimensional taxonomy for conceptualizing and operationalizing the various…

An analysis was made of the cognitive processes that support the judgments made in standard setting activities. These processes were conceived as having two components: forming the domain needed to pass the test and identifying the criterion level of performance to pass the test. In fact, these processes are interactive, and were separated for the…