In this research, we investigated the suitability of implementing "e-rater"® automated essay scoring in a high-stakes large-scale English language testing program. We examined the effectiveness of generic scoring and 2 variants of prompt-based scoring approaches. Effectiveness was evaluated on a number of dimensions, including agreement…

While much of the work in language testing is concerned with constructing quality tests in order to measure language knowledge in reliable and valid ways, there has been a significant movement in language testing research that examines tests in the context of their use in education and society. This line of research exits from the notion that…

This study examines the practical implications of using state tests to measure student achievement in impact evaluations that span multiple states and grades. In particular, the study examines the sensitivity of impact findings to (1) the type of assessment used to measured achievement (state tests or an external assessment administered by the…

A number of brief papers are presented to provide guidelines for test directors of school systems. This collection is intended for both newly appointed and experienced directors. Contributions were solicited from practicing directors of testing; the authors include Anthony J. Allen, Margaret Backman, Joan Bollenbacker, Gerald Hanna, James Lawson;…

The Maryland School Performance Assessment Program (MSPAP) is a relatively new, statewide performance assessment of students in grades 3, 5, and 8. When first administered in May of 1991, the MSPAP included a battery of performance assessment tasks designed to generate written or drawn responses to reading texts. This study evaluated selected…

A standard-setting procedure was developed for the Georgia Teacher Certification Testing Program as tests in 30 teaching fields were revised. A list of important characteristics of a standard-setting procedure was derived, drawing on the work of R. A. Berk (1986). The best method was found to be a highly formalized judgmental, empirical Angoff…

Many approaches to standard-setting use item calibration and student score estimation results to structure panelists' tasks. However, this requires collecting standard-setting judgments after the item analysis results are available. The Scoring Guide Alignment approach collects standard-setting judgments during the scoring sessions from teachers…

This study evaluated agreement between expert system and human scores on 12 algebra word problems taken by Graduate Record Examinations (GRE) General Test examinees from a general sample of 285 and a study sample of 30. Problems were drawn from three content classes (rate x time, work, and interest) and presented in four constructed-response…

This document presents information to trainers already trained in the Registered Holistic scoring method. It has been developed to assist these trainers in training others in the procedures and the criteria of the new scoring method. The Registered Holistic scoring method was designed and field tested for the ninth grade writing sample component…

The Palos Verdes Peninsula Unified School District Office of Program Evaluation and Research is responsible for providing information for program development and improvement; providing test information to special programs coordinators; and acting as a clearinghouse for all information concerning tests, evaluation methodology, and educational…

The methods used by the General Educational Development Testing Service (GEDTS) to establish and maintain score stability and reading reliability on its direct assessment of writing are described. Using the 1988 site certification and monitoring results of several scoring sites, the focus is on describing how the score scale was established and…

Proposes a process approach to validity that addresses assessment validation in the context of high-stakes assessment. This approach includes a test evaluator or validator who considers the perspectives of five stakeholder groups at four different stages of assessment maturity in relation to six aspects of construct validity. Illustrates each…

In view of social processes, politics and the logistics of conducting evaluations, several implicit assumptions of the field challenged by findings on innovation implementation are reviewed. These studies have major implications for evaluation practice, evaluator roles, the use of evaluation results, and the federal role in evaluation. (Author/CM)

Focuses on the process of evaluation capacity building (ECB) in a school district from the perspective of an internal evaluator hired to develop the evaluation function of the district. Shows how the work was originally conceived and why it changed. Emphasizes the core function of the district's Assessment Department, managing the standardized…

A technical advisory committee assisted the Alabama State Department of Education in establishing passing scores and equating test forms of the Alabama High School Graduation Examination (AHSGE). The committee examined four methods of setting cutting scores: (1) Jensen's theoretical method--probability theory is used to set standards that are…