The number of performance assessments continues to increase around the world, and it is important to explore new methods for evaluating the quality of ratings obtained from raters. This study describes an unfolding model for examining rater accuracy. Accuracy is defined as the difference between observed and expert ratings. Dichotomous accuracy…

Attempted to improve the measurement quality of a complex performance assessment through principled assessment design using the example of the National Board for Professional Teaching Standards Early Childhood/Generalist examination. All indexes examined improved after revisions were made. Results show the importance of attention to assessment…

How common patterns of rater errors may be detected in a large-scale performance assessment setting is discussed. Common rater effects are identified, and a scaling method that can be used to detect them in operational data sets is presented. Simulated data sets are generated to exhibit each of these rater effects. The three continua that depict…

Unstable, and potentially invalid, variance component estimates may result from using only a limited portion of available data from operational performance assessments. However, missing observations are common in these settings because of the nature of the assessment design. This paper describes a procedure for overcoming the computational and…

The amount of variability contributed to large-scale performance assessment scores by raters is a constant concern for those who wish to use results from these assessments for educational decisions. This study approaches the problem by examining the behaviors of essay scorers who demonstrate different levels of proficiency with a holistic scoring…

Barriers to large-scale portfolio assessment were studied by surveying 206 secondary teachers interested in adopting these forms of assessment. A rating scale model based on the Rasch model was used to analyze results. Suggestions are presented for facilitating the efforts of secondary school teachers. (SLD)

This paper reports the results of an analysis of the relationship between scorer behaviors and score variability. Thirty-six essay scorers were interviewed and asked to perform a think-aloud task as they scored 24 essays. Each comment made by a scorer was coded according to its content focus (i.e. appearance, assignment, mechanics, communication,…

Describes the design of four science-performance tasks for grade 9 students and the relationship between their performance on those tasks and multiple-choice items on the Iowa Tests of Educational Development. The students and schools used to develop the tasks were not included in the verification sample. Contains 22 references. (Author/ASK)

This paper presents a model of scored cognition that incorporates two types of mental models: models of performance (i.e., the criteria for judging performance) and models of scoring (i.e., the procedural scripts for scoring an essay). In Study 1, six novice and five experienced scorers wrote definitions of three levels of a 6-point holistic…

This paper reports the results of a large-scale portfolio pilot in which over 2,000 secondary students submitted portfolios in language arts, mathematics, and science classes. Students were asked to select work for their portfolios based on the criteria that would be used to evaluate the work. Students also reflected on how their work satisfied…