This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

In the current study, two pools of 250 essays, all written as a response to the same prompt, were rated by two groups of raters (14 or 15 raters per group), thereby providing an approximation to the essay's true score. An automated essay scoring (AES) system was trained on the datasets and then scored the essays using a cross-validation scheme. By…

This article discusses critical methodological design decisions for collecting, interpreting, and synthesizing empirical evidence during the design, deployment, and operational quality-control phases for automated scoring systems. The discussion is inspired by work on operational large-scale systems for automated essay scoring but many of the…

Many innovative item formats have been proposed over the past decade, but little empirical research has been conducted on their measurement properties. This study examines the reliability, efficiency, and construct validity of two innovative item formats--the figural response (FR) and constructed response (CR) formats used in a K-12 computerized…

Item response theory (IRT) methods are generally used to create score scales for large-scale tests. Research has shown that IRT scales are stable across groups and over time. Most studies have focused on items that are dichotomously scored. Now Rasch and other IRT models are used to create scales for tests that include polytomously scored items.…

Using S. J. Messick's theoretical work concerning construct validity as a guide, underlying hypotheses for investigation when validating placement test decisions are assessed. Guidelines on validating placement decisions are offered, and the hypotheses and guidelines are applied in a validation study of the Written English Expression Placement…

Computerized-adaptive (CAT) and self-adapted (SAT) music listening tests were compared for efficiency, reliability, validity, and motivational benefits with 53 junior high school students. Results demonstrate trade-offs, with greater potential motivational benefits for SAT and greater efficiency for CAT. SAT elicited more favorable responses from…

This article explores the measurement problems associated with granting accommodations for mental disabilities, uses existing case law to construct a legal framework for considering such accommodations, and discusses the advantages and disadvantages of alternative strategies for handling testing accommodation requests. (Author/SLD)

A method to investigate consequential evidence of validity for a state assessment developed to change teacher instructional practices is presented. Survey responses from over 1,000 Kansas teachers were used to construct a path model that allowed effects of the state assessment to be studied at building and teacher levels. (SLD)

In low-stakes testing, the motivation levels of examinees are often a matter of concern to test givers because a lack of examinee effort represents a direct threat to the validity of the test data. This study investigated the use of response time to assess the amount of examinee effort received by individual test items. In 2 studies, it was found…

Federal law requires that individuals with handicapping conditions be administered assessments in ways that accommodate their disabilities without penalizing them. Validation studies are needed to evaluate the meaning of scores resulting from nonstandard test administrations. The limited number of these studies to date is reviewed. (SLD)

Measures of linguistic competence for limited-English-proficient students are discussed. The results for 134 students in grades 3 through 6 from a study of the reliability and validity of the Sentence Verification Technique tests as measures of listening and reading comprehension performance in native languages and English are reported. (TJH)

Teacher testing-related attitudes and practices related to court-ordered achievement testing were studied through a mail survey completed by 79 elementary school teachers in a midwestern urban district. Teachers engaged in a large number of test preparation practices and reported finding minimal value in purpose or results of testing. (SLD)