Studied the use of descriptive graphic rating scales by 11 raters to evaluate students' work, exploring different design features. Used a Rasch-model based rating scale analysis to determine that all the continuous scales could be considered to have at least five points, and that defined midpoints did not result in higher student separation…

Examined a procedure for identifying and resolving discrepancies in ratings, focusing on the third rater adjudication procedure used in scoring the Test of Spoken English. Results for 1,446 adult examinees demonstrate that implementing a discrepancy resolution procedure is not sufficient in itself for quality control monitoring. (SLD)

Describes a class of rater effects, differential rater functioning over time (DRIFT), that depicts rater-by-time interactions. Also describes Rasch measurement procedures designed to identify these types of DRIFT in rating data. Applied these procedures to simulated data to show their usefulness in classifying raters as aberrant or non-aberrant…

This paper describes a class of rater effects that depict rater-by-time interactions. This class of rater effects is referred to as differential rater functioning over time (DRIFT). This article describes several types of DRIFT (primacy/recency, differential centrality/extremism, and practice/fatigue) and Rasch measurement procedures designed to…

Developing scoring rubrics to evaluate student work was studied, concentrating on the use of intermediate points in rating scales. How scales that allow for intermediate points between defined categories should be constructed and used was explored. In the recent National Assessment of Educational Progress (NAEP) visual arts field test, researchers…

Analyses of 10 raters' reasoning during think-aloud interviews provided evidence to support a model of the fundamental processes involved in rating standards-based, nonprescriptive portfolios. This process model provides a framework within which to conceptualize sound-rater reasoning and to identify reasoning that distorts the meaning of scores.…

The aesthetic judgments of experts (casting directors and high school drama teachers), theater buffs, and novices were compared as they rated high school students' videotaped performances of Shakespearean monologues. It was hypothesized that theater buffs would represent an intermediate stage on the path to developing expertise in judging acting…

The aesthetic judgments of experts (casting directors and high school drama teachers), theater buffs, and novices were compared as they rated the videotaped performances of high school students performing Shakespearean monologues. Focus was on going beyond the determination of between-judge agreement to determine whether there were objective…