Human raters' assessment of interpreting is a complex process. Previous researchers have mainly relied on verbal reports to examine this process. To advance our understanding, we conducted an empirical study, collecting raters' eye-movement and retrospection data in a computerised interpreting assessment in which three groups of raters (n = 35)…

Researchers have documented the impact of rater effects, or raters' tendencies to give different ratings than would be expected given examinee achievement levels, in performance assessments. However, the degree to which rater effects influence person fit, or the reasonableness of test-takers' achievement estimates given their response patterns,…

Assessing integrated reading-into-writing task performances is known to be challenging, and analytic rating scales have been found to better facilitate the scoring of these performances than other common types of rating scales. However, little is known about how specific operationalizations of the reading-into-writing construct in analytic rating…

Researchers have expressed concern about "implementation" fidelity in intervention research but have not extended that concern to "assessment" fidelity, or the extent to which pre-/posttests are administered and interpreted as intended. When studying reading interventions, data gathering heavily influences the identification of…

Background: Professional standards for validity of achievement tests have long reflected a consensus that validity is the degree to which evidence and theory support interpretations of test scores entailed by the intended uses of tests. Yet there are convincing lines of evidence that the standards are not adequately followed in practice, that…

Ways to use standard-setting data to adjust cutoff scores on examinations are reviewed. Ten sources of information to be used in determining standards are listed. The decision to modify passing scores should be based on these types of information and consideration of adverse impact or rating process irregularities. (SLD)

The purposes of this study were, first, to gather descriptive information regarding the measurement and evaluation skills actually utilized by junior high school teachers and, second, to identify differences among elementary, junior, and senior high school teachers. A questionnaire of 41 statements assessed on a five-point scale the usefulness to…

Possible sources of intrajudge inconsistency in standard setting are reviewed, and approaches are presented to improve the accuracy of rating. Procedures for providing judges with feedback through discussion or computerized communication are discussed. Monitoring and maintaining judges' consistency throughout the rating process are essential. (SLD)

Several issues are related to the use of educational tests. First, test users must be able to choose appropriate tests, interpret scores, and make decisions based on scores. In the field of educational testing, few test users have adequate training in these areas. Second, test makers must clearly specify directions for administration, allowable…

An approach is presented to the definition of minimal competence for judges to use in standard setting. Panelists in standard setting must receive training to ensure that differences in rating result from differences in perceptions of item difficulty, not in differences of opinion about the definition of minimal competence. (SLD)

This manual was prepared to assist in the development of skills requisite to rating the performance of student or beginning teachers. The activities prescribed in the manual are intended to enable experienced teachers to describe the spectrum of performances indicative of the 18 competencies subsumed in the Teacher Performance Assessment…

Issues concerning the selection of judges for standard setting are discussed. Determining the consistency of judges' recommendations, or their congruity with other expert recommendations, would help in selection. Enough judges must be chosen to allow estimation of recommendations by an entire population of judges. (SLD)

Training judges to generate item ratings in standard setting once the reference group has been defined is discussed. It is proposed that sensitivity to the factors that determine difficulty can be improved through training. Three criteria for determining when training is sufficient are offered. (SLD)

A number of brief papers are presented to provide guidelines for test directors of school systems. This collection is intended for both newly appointed and experienced directors. Contributions were solicited from practicing directors of testing; the authors include Anthony J. Allen, Margaret Backman, Joan Bollenbacker, Gerald Hanna, James Lawson;…

The effects of using recommended data collection procedures on median recommended test standards, variability of recommended test standards, and reliability of recommended standards for 7 subtests of the National Teacher Examinations Communications Skills and General Knowledge Tests were explored, using 236 evaluators (75 public school teachers…