This article discusses critical methodological design decisions for collecting, interpreting, and synthesizing empirical evidence during the design, deployment, and operational quality-control phases for automated scoring systems. The discussion is inspired by work on operational large-scale systems for automated essay scoring but many of the…

Test validity refers to the degree to which the inferences based on test scores are meaningful, useful, and appropriate. Thus, test validity is a characteristic of a test when it is administered to a particular population. This article introduces the modern concepts of validity advanced by S. Messick (1989, 1996, 1996). Traditionally, the means of…

Candidates who had taken examinations for certification required by the American Production and Inventory Control Society (APICS) were surveyed to acquire feedback about the effectiveness of score interpretive information given to test takers. Those sampled included 488 passers and 389 failers of the Inventory Management (IM) examination and 457…

The goal of the No Child Left Behind Act (NCLB) is to have 100 percent of America's public school students "proficient" by the year 2014. As Rosenberg points out, however, there are multiple interpretations of proficiency, including: (1) Different methods of setting performance standards yield different meanings of proficient; (2)…

Students with severe/profound mental retardation performed functional tasks under training or probe conditions. Analysis of nine students' records indicated that students do not react to probe conditions as if they were extinction conditions; training data do not consistently reflect performance under probe conditions; and data collected under…

The use of reporting final examinations in the Netherlands is described. Each October a public report is published by Cito, supplying information about the achieved level of education and describing observed psychometric qualities of the measurement instrument. General information needs are provided by the National Bureau of Statistics. (LMO)

When the Educational Testing Service became the administrator of the National Assessment of Educational Progress (NAEP) in 1983, it introduced scales based on item response theory (IRT) as a way of presenting results of the assessment to the general public. Some properties of the scales and their uses are discussed. Initial attempts at presenting…

This report analyzes possible causes of current trends in educational achievement and discusses implications for policy. It uses data described in the publication "Trends in Educational Achievement," a Congressional Budget Office study released in April 1986, which assessed data about trends in test scores. This report begins with a…

Calls for simplicity when presenting data on student achievement. Data should help teachers improve teaching and learning, and focus on specific goals such as determining how many students are succeeding in a subject and, within that subject, what are the areas of strength or weakness. (Contains 22 references.) (WFA)

Too few researchers understand what statistical significance testing does and does not do, and consequently their results are misinterpreted. This Digest explains the concept of statistical significance testing and discusses the meaning of probabilities, the concept of statistical significance, arguments against significance testing,…

I discuss the contributions of Harvey Siegel, Francis Schrag and Randall Curren to this volume. Their articles cast in bold relief the relation of High Stakes Testing to the goals of education, the nature of mind and the demands of justice. I argue that the connections are deep, but that the considerations these authors raise do not show that High…

Discusses which kinds of data educators should respect and which they should reject. Asserts that most state accountability tests fail to produce the kinds of data that will improve teaching and learning. Teachers can get the data they need from their own instructionally useful classroom assessments. (WFA)

The paper focuses on three critical programming issues in providing support services to higher education students with learning disabilities. The first issue concerns the garnering of administrative support for students with learning disabilities. Administrative support cannot be effectively achieved until college administrators determine where…