ERIC Number: ED290795
Record Type: Non-Journal
Publication Date: 1987-Apr
Pages: 9
Abstractor: N/A
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Effects of Rating Task Instructions on Consistency and Accuracy of Expert Raters.
Littlefield, John H.; Troendle, G. Roger
The effect of different types of rating task instructions on rater behavior was examined using experts, as opposed to novices, as raters. The experts were instructed to (1) form a global categorical judgment (early hypothesis generation); (2) assess 19 detailed elements; or (3) both. Subjects were 8 dental faculty members who ranged in age from 28 to 60 and who had at least 2 years of teaching experience. The task was to evaluate five three-fourth dental crown preparations twice, using each of the three types of rating instructions. Intrarater and interrater agreement and reliability were assessed, as was the level of rater accuracy. Higher coefficients of rater reliability, but not agreement, resulted from the global and combined instructions, compared with the 19-point, or traditional, instruction. The global judgment alone improved reliability over traditional instructions, but intrarater agreement was lower. Expert rater consistency was higher when early hypothesis generation and self-monitoring were encouraged by the rating instructions. There were no significant differences in score accuracy among the three types of instruction. (MGD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A