Researchers have explored a variety of topics related to identifying and distinguishing among specific types of rater effects, as well as the implications of different types of incomplete data collection designs for rater-mediated assessments. In this study, we used simulated data to examine the sensitivity of latent trait model indicators of…

Range restrictions, or raters' tendency to limit their ratings to a subset of available rating scale categories, are well documented in large-scale teacher evaluation systems based on principal observations. When these restrictions occur, the ratings observed during operational teacher evaluations are limited to a subset of the available…

Even though guessing biases difficulty estimates as a function of item difficulty in the dichotomous Rasch model, assessment programs with tests which include multiple-choice items often construct scales using this model. Research has shown that when all items are multiple-choice, this bias can largely be eliminated. However, many assessments have…

Equating methods make use of an appropriate transformation function to map the scores of one test form into the scale of another so that scores are comparable and can be used interchangeably. The equating literature shows that the ways of judging the success of an equating (i.e., the score transformation) might differ depending on the adopted…

Rating scale items have been widely used in educational and psychological tests. These items require people to make subjective judgments, and these subjective judgments usually involve randomness. To account for this randomness, Wang, Wilson, and Shih proposed the random-effect rating scale model in which the threshold parameters are treated as…

Orlando and Thissen's S-X[superscript 2] item fit index has performed better than traditional item fit statistics such as Yen' s Q[subscript 1] and McKinley and Mill' s G[superscript 2] for dichotomous item response theory (IRT) models. This study extends the utility of S-X[superscript 2] to polytomous IRT models, including the generalized partial…

The influence of the verbal label and its scalar position in defining the meaning of the labeled position on a rating scale was studied in three forms of the scale with the labels FAIR and GOOD systematically moved. When labels and position differed in meaning, college students rated the labeled position as a compromise between the two. (SLD)

A coefficient of inter-rater agreement is presented which describes the magnitude of observer agreement as the probability estimated under a quasi-independence model that responses from different observers will be in agreement. (Author/JKS)

Information loss occurs when continuous data are grouped in discrete intervals. After calculating the squared correlation coefficients between continuous data and corresponding grouped data for four population distributions, the effects of population distribution, number of intervals, and interval width on information loss and recovery were…

The use of matrix sampling to overcome the impracticality of the pair comparison method for obtaining scale values is empirically tested. Results indicate that matrix sampling is useful in such applications. (Author/JKS)

For measuring the face validity of a test, Nevo suggested that test takers and nonprofessional users rate items on a five point scale. This article questions the ability of those raters and the credibility of the aggregated judgment as evidence of the validity of the test. (JAZ)

The relationship between naturally-occurring student expectations about the instructor and later student evaluations of that instructor was studied. It was found that students were capable of rating their instructors independently of expectations held prior to the course. (Author/BW)

Ratings given to questionnaire items on four types of rating scales were compared. This study shows that the differences between scales are contingent upon the particular anchors used for the intermediate options. The results suggest that the mean score is predictably influenced by changes in the intermediate anchors. (Author/PN)