The weighted kappa coefficient is a measure of interrater agreement when the relative seriousness of each possible disagreement can be quantified. This monte carlo study demonstrates the utility of the kappa coefficient for ordinal data. Sample size is also briefly discussed. (Author/JKS)

An alternative formulation for Guttman scaling is presented. The new formulation is described, and advantages over Guttman's formulation are detailed. The method is assumption-free and capable of multidimensional analysis. (Author/JKS)

D.V. Budescu and J.L. Rogers (1981) proposed a method of adjusting correlations of scales to eliminate spurious components resulting from the overlapping of scales. Three reliability correction formulas are derived in this article that are based on more tenable assumptions. (SLD)

A least-squares solution for the method of paired comparisons is given. The approach provokes a theorem regarding the amount of data necessary and sufficient for a solution to be obtained. A measure of the internal consistency of the least-squares fit is developed. (Author/CTM)

Joreskog's model of cogeneric tests is used to analyze agreement between raters. Raters are treated as measuring instruments. The model of cogeneric tests, of which classical parallelism and tau-equivalence are shown to be special cases, is applied to teachers' ratings of students' responses on open-end questions. (Author/RC)

The psychometric objections to using essays and other subjective tests for measurement can be overcome by a many-facet Rasch model. This model enables judge-awarded grades to be transformed from their arbitrary, local, non-linear rating scale form into linear measures with explicit generalizable meaning of specifiable reliability (standard error)…

This annotated bibliography was developed upon noting a deficiency of information in the literature regarding the training of raters for establishing agreement. The ERIC descriptor, "Interrater Reliability", was used to locate journal articles. Some of the 33 resulting articles focus on mathematical concepts and present formulas for computing…

A procedure is presented for the analysis of rating data with correlated intrajudge and uncorrelated interjudge measurement errors. Correlations between true scores on different rating dimensions, reliabilities for each judge on each dimension and correlations between intrajudge errors can be estimated given a minimum of three raters and two…

Three frequently used response formats are compared via analysis of covariance structures. The cumulative results based on four data sets provided evidence inconsistent with previous research suggesting that these formats are interchangeable. The semantic-differential format is most preferred while in most cases the Stapel format is least…

Three correlation matrices (PEARSON, POLYCHORIC, and TOBIT) were used to obtain reliability estimates on ordered polytomous variable models. A Monte Carlo study with different levels of variable asymmetry and 400 sample correlation matrices demonstrated that the PEARSON matrix did not perform as well as did the other 2 matrices. (SLD)

Prior research in a single clinical training setting has shown Cason and Cason's (1981) simplified model of their performance rating theory can improve rating reliability and validity through statistical control of rater stringency error. Here, the model was applied to clinical performance ratings of 14 cohorts (about 250 students and 200 raters)…

Generalizability theory extends previous methods of estimating the reliability of rating instruments such that one can estimate the precision of a measurement system for differentiating among students, scales, or any other important dimension. In this study, generalizability theory is applied to faculty ratings of junior and senior dental students…