This paper reports the results of an analysis of the relationship between scorer behaviors and score variability. Thirty-six essay scorers were interviewed and asked to perform a think-aloud task as they scored 24 essays. Each comment made by a scorer was coded according to its content focus (i.e. appearance, assignment, mechanics, communication,…

Students at grades four and five were administered a writing assessment that was developed to correspond to the California Learning Assessment System (CLAS) writing tasks at grade four. Teachers were trained to score the CLAS-like tasks according to the rubric developed by the State for CLAS. In addition, 164 students at three schools in the…

Writing fits well within the realm of outcomes suitable for observation by performance assessments. Studies of the reliability of performance assessments have suggested that interrater reliability can be consistently high. Scoring consistency, however, is only one aspect of quality in decisions based on assessment results. Another is…

Results of the International English Language Testing System (IELTS) battery trials in Australia are reported. The IELTS tests of productive language skills use direct assessment strategies and subjective scoring according to detailed guidelines. The receptive skills tests use indirect assessment strategies and clerical scoring procedures.…

The effectiveness of an optimal item selection method in designing parallel test forms was studied during the development of two forms that were parallel to an existing form for each of three language arts tests for fourth graders used in the Connecticut Mastery Test. Two listening comprehension forms, two reading comprehension forms, and two…

Across the field of educational assessment the case has been made for alternatives to the multiple-choice item type. Most of the alternative types of items require a subjective evaluation by a rater. The reliability of this subjective rating is a key component of these types of alternative items. In this paper, measures of reliability are…

In the Dutch Educational Assessment Program, the students' language proficiency is measured in grade 9, at age 15. Writing performance is measured through several performance-based writing tasks, rated on numerous aspects such as content, style, organization, punctuation, spelling and grammar. As a consequence, national performance levels are…

The scaling of a new assessment is a significant undertaking. The scaling of a new assessment designed as a multiple-level, criterion-referenced assessment is even more so. A Guttman approach to scaling was used with the Work Keys selected-response assessments, Reading for Information and Applied Mathematics. Assessments in development in the Work…