This article presents a comparative judgment approach for holistically scored constructed response tasks. In this approach, the grader rank orders (rather than rate) the quality of a small set of responses. A prior automated evaluation of responses guides both set formation and scaling of rankings. Sets are formed to have similar prior scores and…

In a recent Monte Carlo simulation study, Liu and Zumbo showed that outliers can severely inflate the estimates of Cronbach's coefficient alpha for continuous item response data--visual analogue response format. Little, however, is known about the effect of outliers for ordinal item response data--also commonly referred to as Likert, Likert-type,…

In an effort to standardize academic application procedures, the authors developed the Standardized Letters of Recommendation (SLR) to capture important cognitive and noncognitive qualities of graduate school candidates. The SLR, which consists of seven scales, is applied to an intern-selection scenario. Both professor ratings (n = 414) during the…

This study examined the reliability and validity of the scores of a Chinese-translated version of the Gifted Rating Scales-School Form (GRS-S) using parents as raters and explored the effects of gender and grade on the ratings. A total of 222 parents participated in the study and rated their child independently using the Chinese version of the…

A computer program entitled Program CONGRU is presented which analyzes the overall conjoint agreement among many observers for their classifications on categorical scales. A special feature assesses the significance of observers' congruence in assignments to each nominal category considered. (Author)

Given judges' ratings on a set of variables, the program will produce for each variable, a printed analysis of variance summary table for between subjects/events, within subjects/events, and between judges sources of variability, a reliability coefficient, an adjusted reliability coefficient and means and standard deviations for each rater. (RC)

Since both the means and reliabilities of the ratings were affected by the number of scale levels but not by the manner of defining scale levels, it would appear that the most critical consideration in rating scale construction is that of determining the appropriate number of rating scale levels. (Author/MB)

Two experiments were conducted to assess the ability of preservice teachers to score moral thought statements using Kohlberg's theory of moral development. Results indicate that teachers, in general, are not able to rate the statements with consistency. (JKS)

Reliability and construct and criterion-related validity of scores on the Disruptive Behavior Rating Scale--Teacher Version were studied with 151 teachers of 1,766 elementary school students and 30 teachers of 131 students. Factor analysis confirmed a four-factor structure. Results support the measure's internal consistency and convergent…

Stable factors of student ratings of teachers have been found under a variety of settings. The author suggests that these factors are artifacts of the raters' implicit categories of teaching behaviors and are not necessarily reflective of the behaviors themselves. (JKS)