The materials in this paper were drawn from an action caucus called to seek common criteria for the presentation and judging of oral interpretation and forensic tournaments. The first section of the paper consists of presentations made by participants. Topics covered in these presentations are (1) consistency in judging, (2) the role of oral…

Four experienced stuttering researchers viewed videodisks of spontaneous speech from chronic stutterers and attempted to locate the precise onset and offset of individual stuttering events. Results showed interjudge disagreements that challenge the reliability and validity of onset and offset judgments. Highly agreed stuttering events were…

This study compares intra- and inter-rater agreement and reliability when using three different rating form formats to assess the same stimuli. One format requests assessment by marking detailed criteria without an overall judgement; the second format requests only an overall judgement without the use of detailed criteria; and the third format…

This paper reports on the investigation of the operation of the Teacher of the Year Award program in 28 states. It focuses on the method of selection for recipients of this award, and explores the nature of the judging process used in selecting candidates at both the state and national levels. Specific objectives are: (1) to determine the criteria…

In circumstances where judges are required to make ratings of performance, it is usually required to have two or more raters who are trained to agree on independent ratings of the same performance. It is suggested that such a requirement may produce the paradox of attenuation associated with item analysis, in which too high a correlation between…

Examines how judicial selection methods (appointive or elective), electoral power, and intergroup competition influence number of state judgeships among women, Blacks, and Hispanics. Reports the impact of selection methods has been overstated. Shows a positive relationship between number of women and minority lawyers and number of women and…

This study used generalizability theory and other statistical concepts to assess the application of the Angoff method to setting cutoff scores on two professional certification tests. A panel of ten judges gave pre- and post-feedback Angoff probability ratings of items of two forms of a professional certification test, and another panel of nine…

This study examined the contribution of five categories of variables to the standard-setting recommendations of a representative sample of 241 judges recommending standards for the General Knowledge and Communication Skills Tests of the National Teacher Examinations (NTE) for the state of North Carolina. It was based on the assumption that…

Minimum competency cut-off scores on a statistics exam were estimated under four conditions: the Angoff judging method with item data (n=20), and without data available (n=19); and the Modified Angoff method with (n=19), and without (n=19) item data available to judges. The Angoff method required free response percentage estimates (0-100) percent,…

The research described in this paper examines the relationship between raters and the training raters receive to apply that rating. The primary concern is maintaining scoring consistency within the context of raters' perceptions. If the criteria for holistic and/or analytic scoring can be recognized as communication based, rather than topic based,…

This study explores the use of the modified caution index (MCI) for identifying judges whose patterns of recommendations suggest that their judgments might be based on incomplete information, flawed reasoning, or inattention to their standard-setting tasks. It also examines the effect on test standards and passing rates when the test standards of…