Drawing upon archival materials, I describe the history, design, and assessment of literacy tests from early 20th century New York state. Practitioners working with these early standardized writing tests grappled with tensions created by public Nativist sentiment, the legislation of "literacy," and calls to score the tests in…

This article reviews key developmental theories that have been adopted by writing development researchers over the last fifty years. It describes how researchers have translated these theories into definitions of writing development capable of influencing curricular design and interpretations of student writing and explores the implications for…

Writing task variants can increase test security in high-stakes essay assessments by substantially increasing the pool of available writing stimuli and by making the specific writing task less predictable. A given prompt (parent) may be used as the basis for one or more different variants. Six variant types based on argument essay prompts from a…

Composition assessment scholars have exhibited uneasiness with the language of norming grounded in distaste for the psychometric assumption that achievement of consensus in a communal assessment setting is desirable even at the cost of individual pedagogical values. Responding to the problems of a "reliability" defined by homogenous agreement,…

In this paper, we provide an overview of psychometric procedures and guidelines Educational Testing Service (ETS) uses to evaluate automated essay scoring for operational use. We briefly describe the e-rater system, the procedures and criteria used to evaluate e-rater, implications for a range of potential uses of e-rater, and directions for…

This article discusses three sets of challenges involved in the assessment of writing from a developmental perspective. These challenges include defining a workable theory of development, developing a suitable construct, and overcoming limitations in technocentric approaches to writing assessment. In North America in recent years, a burgeoning…

This article offers one potential response to Yancey's (1999) call for a fourth wave of writing assessment able to capture writing development in all of its complexity. Based on an ecological perspective of literate development that situates students' growth as writers across multiple engagements with writing, including those outside of school,…

Raters of Georgia's (USA) state-mandated college-level writing exam, which is intended to ensure a minimal university-level writing competency, are trained to grade holistically when assessing these exams. A guiding principle in holistic grading is to not focus exclusively on any one aspect of writing but rather to give equal weight to style,…

This paper examines the construct measured by automated essay scoring (AES) systems. AES systems measure features of the text structure, linguistic structure, and conventional print form of essays; as such, the systems primarily measure text production skills. In the current state-of-the-art, AES provide little direct evidence about such matters…

In this paper, I describe the design and evaluation of automated essay scoring (AES) models for an institution's writing placement program. Information was gathered on admitted student writing performance at a science and technology research university in the northeastern United States. Under timed conditions, first-year students (N = 879) were…

Rating scales act as the de facto test construct in a writing assessment, although inevitably as a simplification of the construct (North, 2003). However, it is often not reported how rating scales are constructed. Unless the underlying framework of a rating scale takes some account of linguistic theory and research in the definition of…

This study investigated the use of automated essay scoring (AES) to identify at-risk students enrolled in a first-year university writing course. An application of AES, the "Criterion"[R] Online Writing Evaluation Service was evaluated through a methodology focusing on construct modelling, response processes, disaggregation, extrapolation,…

Automated Essay Scoring (AES) has garnered a great deal of attention from the rhetoric and composition/writing studies community since the Educational Testing Service began using e-rater[R] and the "Criterion"[R] Online Writing Evaluation Service as products in scoring writing tests, and most of the responses have been negative. While the…

This article presents considerations for using automated scoring systems to evaluate second language writing. A distinction is made between English language learners in English-medium educational systems and those studying English in their own countries for a variety of purposes, and between learning-to-write and writing-to-learn in a second…

This paper describes the process by which a rubric development team affiliated with the National Writing Project negotiated difficulties and dilemmas concerning an analytic scoring category initially termed Voice and later renamed Stance. Although these labels reference an aspect of student writing that many teachers value, the challenge of…