In this paper, we provide an overview of psychometric procedures and guidelines Educational Testing Service (ETS) uses to evaluate automated essay scoring for operational use. We briefly describe the e-rater system, the procedures and criteria used to evaluate e-rater, implications for a range of potential uses of e-rater, and directions for…

Automated Essay Scoring (AES) has garnered a great deal of attention from the rhetoric and composition/writing studies community since the Educational Testing Service began using e-rater[R] and the "Criterion"[R] Online Writing Evaluation Service as products in scoring writing tests, and most of the responses have been negative. While the…

Writing is complex, and assessment of writing is equally complex, particularly when considering the need to measure outcomes at the institutional level while providing meaningful data that informs curriculum reform and supports learning at the discipline-level. Using a multi-layered assessment that incorporates standardized measures of writing…

Mixed quantitative and qualitative methods were used to assess student attitudes toward the assessment criteria for higher-order critical and analytical thinking skills in writing-intensive curricula in first-year, general education courses. The courses varied in emphasis on critical thinking in the criteria used to assess writing. The analysis is…

The purpose of this study is to examine the interactions among measurement theories, writing theories, and writing assessments in the United States from an historical perspective. The assessment of writing provides a useful framework for examining how theories influence, and in some cases fail to influence actual practice. Two research traditions…

Establishing the score or the placement as the first priority in a writing assessment leads to more reductive forms of writing assessment. However, if the prompts used in a direct test of writing were generative--that is, if they asked test-takers to analyze their own experiences as writers or learners, for example--the resulting texts would be…

This is the story of a research journey that follows the trail of a novel evaluand--"place." I examine place as mentioned by rising juniors in timed exams. Using a hybridized methodology--the qualitative approach of a hermeneutic dialectic process as described by Guba and Lincoln (1989), and the quantitative evidence of place mention--I query…

We present some key findings of a four-year, two-phase writing assessment project at Central Michigan University: Phase One (2002), a survey of faculty members (n=115) and subsequent focus groups (n=14) and Phase Two (2005), an evaluation of two samples of student writing (n=635 and 632). Major findings of Phase One reported here include the…

It is widely assumed that the constraints of timed essay exams will make it virtually impossible for students to engage in the major hallmarks of the writing process, especially revision, in testing situations. This paper presents the results of a study conducted at Washington State University in the Spring of 2008. The study examined the…

Testing the hypotheses that reflective timed-essay prompts should elicit memories of meaningful experiences in students' undergraduate education, and that computer-mediated classroom experiences should be salient among those memories, a combination of quantitative and qualitative research methods paints a richer, more complex picture than either…

Large-scale writing programs can add value to the traditional timed writing assessment by using aspects of the essays to assess the effectiveness of institutional goals, programs, and curriculums. The "six learning goals" prompt in this study represents an attempt to provide an accurate writing assessment that moves beyond scores. This…

The authors explored the credibility of using informal reading inventories and writing samples for 138 students (K-4) to evaluate the effectiveness of a summer literacy program. Running Records (a measure of a child's reading level) and teacher experience during daily reading instruction were used to estimate the reliability of the more formal…

The category of coherence in rating scales has often been criticized for being vague. Typical descriptors might describe students' writing as having 'a clear progression of ideas' or "lacking logical sequencing." These descriptors inevitably require subjective interpretation on the side of the raters. A number of researchers (Connor & Farmer,…

Elaborates on explorations and arguments prompted by the effort undertaken by the Department of Education through the National Center for Education Statistics to begin developing the National Assessment of College Student Learning (NACSL), which focuses on, among other things, "advanced ability to communicate effectively," as specified…

Describes a project in which researchers work with teachers to increase their knowledge of a specific genre of writing and to use the knowledge in response to student work. Finds that few teachers used the feedback response form or scoring rubric provided to them but that teachers used many of the assessment techniques in instructional design and…