A challenge of large-scale oral communication assessments is to feasibly assess a broad construct that includes interactional competence. One possible approach in addressing this challenge is to use a spoken dialog system (SDS), with the computer acting as a peer to elicit a ratable speech sample. With this aim, an SDS was built and four trained…

This technical assistance paper provides guidance to individuals with test administration responsibilities in adult education programs. These policies apply to the approved assessments that programs may use to report educational gains in compliance with the National Reporting System (NRS) requirements. The NRS is the accountability system for the…

This study examines the practical implications of using state tests to measure student achievement in impact evaluations that span multiple states and grades. In particular, the study examines the sensitivity of impact findings to (1) the type of assessment used to measured achievement (state tests or an external assessment administered by the…

A number of brief papers are presented to provide guidelines for test directors of school systems. This collection is intended for both newly appointed and experienced directors. Contributions were solicited from practicing directors of testing; the authors include Anthony J. Allen, Margaret Backman, Joan Bollenbacker, Gerald Hanna, James Lawson;…

Business schools are often criticized for the inadequate writing skills of their graduates. Improving writing skills involves first understanding the current skill level of students. This research attempts to provide insights into the effectiveness of the current method of assessing writing skills in a school of business at a large regional…

The purpose of the current study was to analyze the relationship between automated essay scoring (AES) and human scoring in order to determine the validity and usefulness of AES for large-scale placement tests. Specifically, a correlational research design was used to examine the correlations between AES performance and human raters' performance.…

The Palos Verdes Peninsula Unified School District Office of Program Evaluation and Research is responsible for providing information for program development and improvement; providing test information to special programs coordinators; and acting as a clearinghouse for all information concerning tests, evaluation methodology, and educational…

Focuses on the process of evaluation capacity building (ECB) in a school district from the perspective of an internal evaluator hired to develop the evaluation function of the district. Shows how the work was originally conceived and why it changed. Emphasizes the core function of the district's Assessment Department, managing the standardized…

Eleven judges representing 11 different geographic regions in the United States participated in a standard-setting session designed to determine the possibility of obtaining interjudge consensus and intrajudge consistency simultaneously. Each judge had experience in the field for which standards were being set. The judges rated 65 multiple-choice…

To help evaluators find the right measurement instrument, this annotated bibliography lists 46 compendia listing, reviewing, or presenting instruments devoted to (1) psychological/behavioral assessment; (2) mental health; (3) physical health; (4) sociological scales; (5) attitudinal scales; (6) observational measures; (7) standardized tests; and…

The necessity of controlling the order in which trained essay raters for a statewide writing assessment program receive student essays was studied. The underlying theoretical question concerns possible rater bias caused by raters reading long strings of essays of homogeneous quality; this problem is usually referred to as context effect or…

A formative evaluation of a pilot review and appeal process for the Maryland Writing Test (MWT) is described. The MWT is a large-scale direct assessment of writing. A three-year pilot phase was to culminate in the implementation of an operational procedure for the review and appeal of scores impacting a pass/fail decision for examinees. The MWT…

Across the field of educational assessment the case has been made for alternatives to the multiple-choice item type. Most of the alternative types of items require a subjective evaluation by a rater. The reliability of this subjective rating is a key component of these types of alternative items. In this paper, measures of reliability are…

In the Dutch national examinations, reading comprehension tests are used for all languages. For the native language, reading comprehension is tested with reading passages and related questions to which the test-taker provides his own response, not choosing from a group of alternatives. One problem encountered in testing with these items is…

The role of expert judges in establishing the content validity of the National Teacher Examinations (NTE) was examined in a detailed study. The NTE are used in 22 states for screening applicants to teacher education programs and/or for screening candidates for initial teacher certification. Such extensive use of the NTE has stimulated 35 recent…