Variation in examiner stringency is a recognised problem in many standardised summative assessments of performance such as the OSCE. The stated strength of the OSCE is that such error might largely balance out over the exam as a whole. This study uses linear mixed models to estimate the impact of different factors (examiner, station, candidate and…

Public administration scholarship has documented the effectiveness of rubrics in assessing student competencies within classrooms and across MPA programs. Rubrics are one of many innovative techniques, however. As we advance pedagogy in public administration, scholars should consider additional grading techniques and evaluate a grading system…

Results suggest that different groups of judges do set similar examination standards when using the same procedure, and that the average of individual judgments does not differ significantly from group consensus judgments. Significant differences were found, however, between the standards set by the two procedures employed. (RC)

Data from two standard-setting exercises were analyzed using the logistic regression model that assumes no variation in severity of raters, and results were compared with those obtained by logistic regression that allowed for severity variation. Results illustrate the importance of taking between-rater differences into account. (SLD)

Standard setting on licensure and certification tests is difficult both to execute and to defend. There may, however, be certain minimum standards for standard setting on which most will be able to agree. One such standard is logical consistency. M. T. Kane (1984, 1986) has suggested an approach to evaluating the logical consistency of one widely…

To study the ways that new ways of sharing responsibility for teacher quality affect grading discourses and practices, more than 900 surveys were collected from student/intern teachers and cooperating teachers and supervisors over a 9-year period. Opinions about pass/fail grading for student/intern teaching were tabulated and analyzed. Additional…

A study was conducted to explore empirically the effect of different definitions of the target examinee on the judgment of panelists setting a passing score. Two cut score studies were done in a school district for the same test within a 6-month period, and different definitions of the target candidate were provided for each study. In October…

The objective of this study was to determine the degree to which recommendations for passing scores, calculated on the basis of a traditional standard-setting methodology, might be affected by the mode (paper versus computer-screen prints) in which test items were presented to standard setting panelists. Results were based on the judgments of 31…

The major forms of evidence that support a standard's credibility are reviewed, and what can be done over time and for different forms of an examination to enhance its comparability in a credentialing setting is outlined. Pass-fail decisions must be consistent to ensure a standard's credibility. (SLD)

Whether anchor item sets varying in difficulty and discrimination affect precision of cutting score equivalents generated through judge rescaling as much as equivalents from score equating was studied with 4 groups of experts and 250 and 1,000 examinees. Results indicate the robustness of judge rescaling and its superiority over equating. (SLD)

The grading standards must be based on society's needs and the availability and ability of students to be trained to fill these needs. (IR)

A survey of 12 constituencies (totaling 2,288 respondents) concerning the National Board of Medical Examiners' (NBME) comprehensive Part I and Part II examinations revealed support for more even distribution of fail rates across examinations, support for a fixed standard, and favorable reaction to setting standards based on review of examination…

Results of the fall 1993 administration of part two of the Medical Council of Canada's Evaluating Examination for 744 candidates provided evidence of the consistency of the pass/fail and cutting score definitions for the objective- structured clinical examination stations used across examiners. These results support the validity of this…

Passing scores for licensure and certification tests are justified by showing that decisions based on the passing score achieve the purposes of the credentialing program while avoiding any serious negative consequences. The passing score is justified to the extent decisions based on it have positive consequences. More specifically, the standard…