Logical estimates of item difficulty made by judges were compared to empirical estimates derived from a test administration. Results indicated substantial correspondence between logical and empirical estimates, and substantial variation among judges. Further, the more elaborate the system used by judges to make estimates, the more accurate the…

The validity of two subjective approaches to judging in synchronized swimming were examined through a multitrait-multimethod matrix. Results indicated that judging panels tended not to differentiate between execution and content scores. (Author/JKS)

Proposes a generalization of Cohen's kappa coefficient (J. Cohen, 1960) to address the problem of accounting for overall chance-corrected interobserver agreement among the multivariate ratings of several judges. The statistic's metric is conventional and in the univariate case it is equivalent to existing extensions of the kappa coefficient to…

Conducted a meta analysis of studies of procedural modifications of the Angoff method of setting cutoff scores. Findings for 38 studies (113 judges) show that common modifications have produced systematic effects on cutoff scores and the degree of consensus among judges. (SLD)

New coefficients of agreement are suggested for the measure of intraclass consistency between observations on two variables. The coefficients are derived from a general coefficient for measuring intraclass dependence in a bivariate analysis context. Various coefficients for the univariate agreement analysis are shown to be cases of the suggested…

In a study involving eight judges, analysis with the FACETS model provides evidence that judges grade differently, whether or not scores correlate well. This outcome suggests that adjustments for differences among judges should be made before student measures are estimated to produce reproducible decisions. (SLD)

A new approach based on Rasch measurement theory is described for examining the quality of ratings from standard-setting judges. Ratings of nine judges for 213 items on a nursing examination show that judges vary in their views of the essential items for nursing certification, with statistically significant variability in the judged essentiality…