Automated essay scoring engines (AESEs) are becoming increasingly popular as an efficient method for performance assessments in writing, including many language assessments that are used worldwide. Before they can be used operationally, AESEs must be "trained" using machine-learning techniques that incorporate human ratings. However, the…

The number of performance assessments continues to increase around the world, and it is important to explore new methods for evaluating the quality of ratings obtained from raters. This study describes an unfolding model for examining rater accuracy. Accuracy is defined as the difference between observed and expert ratings. Dichotomous accuracy…

This report summarizes an empirical study that addresses two related topics within the context of writing assessment--illusory halo and how much unique information is provided by multiple analytic scores. Specifically, we address the issue of whether unique information is provided by analytic scores assigned to student writing, beyond what is…

Historically, research focusing on rater characteristics and rating contexts that enable the assignment of accurate ratings and research focusing on statistical indicators of accurate ratings has been conducted by separate communities of researchers. This study demonstrates how existing latent trait modeling procedures can identify groups of…

This study examined the influence of rater training and scoring context on training time, scoring time, qualifying rate, quality of ratings, and rater perceptions. One hundred twenty raters participated in the study and experienced one of three training contexts: (a) online training in a distributed scoring context, (b) online training in a…

Examines behavioral differences of essay scorers who demonstrate different levels of proficiency. Compares three proficiency groups to identify differences in essay features considered, understandings of the scoring rubric, and decision-making procedures. Finds that scorers with different proficiency levels do not focus on different essay features…

Presents a model of the cognitive frameworks and processes used by essay scorers. Investigates the relationship between a scorer's reading style and proficiency with a scoring rubric. Finds that proficient scorers are better able to withhold judgment and may focus their efforts more intensely on the process of evaluation. Discusses implications of…

Compares essays composed by secondary school students with pen and paper to those composed on a word processor. Finds that word-processed essays were neater, longer, more formal, and had a weaker voice than pen-and-paper essays. Finds word-processed essay quality was unaffected for students with high to medium computer experience but was adversely…

This paper presents a model of scored cognition that incorporates two types of mental models: models of performance (i.e., the criteria for judging performance) and models of scoring (i.e., the procedural scripts for scoring an essay). In Study 1, six novice and five experienced scorers wrote definitions of three levels of a 6-point holistic…

Investigates how word processing experience influences tenth grade student performance on a writing assessment. Examines factors influencing a student's decision about using word processors for writing; relationship between experience with the technology and scores on word processed essays; and differences in length, neatness, mechanical…

The Test of English as a Foreign Language (TOEFL) contains a direct writing assessment, and examinees are given the option of composing their responses at a computer terminal using a keyboard or composing their responses in handwriting. This study sought to determine whether performance on a direct writing assessment is comparable for examinees…

This study is an interim follow-up to an investigation of how computer use influences the writing process used by high school students on a direct writing assessment (Wolfe and others, 1994). Whether students with less comfort and experience with word processors would receive lower scores on word-processed essays than those with more comfort and…