In large-scale educational assessment data consistent with a simple-structure multidimensional item response theory (MIRT) model, where every item measures only one latent variable, separate unidimensional item response theory (UIRT) models for each latent variable are often calibrated for practical reasons. While this approach can be valid for…

The National Assessment of Educational Progress (NAEP) -- often called "The Nation's Report Card" -- is the largest nationally representative and continuing assessment of what students in public and private schools in the United States know and can do in various subjects and has provided policy makers and the public with invaluable…

This article considers the claim that machine scoring of writing test responses agrees with human readers as much as humans agree with other humans. These claims about the reliability of machine scoring of writing are usually based on specific and constrained writing tasks, and there is reason for asking whether machine scoring of writing requires…

This report presents results of the 2011 National Assessment of Educational Progress (NAEP) in writing at grades 8 and 12. In this new national writing assessment sample, 24,100 eighth-graders and 28,100 twelfth-graders engaged with writing tasks and composed their responses on computer. The assessment tasks reflected writing situations common to…

Reviewed methods of estimating theta suitable for computerized adaptive testing (CAT) and discussed the differences between Fisher and Kullback-Leibler information criteria for selecting items. Examined the accuracy of different CAT algorithms using samples from the National Assessment of Educational Progress. Results show when correcting for…

A study was conducted to extend the sequential probability ratio testing (SPRT) procedure with the polytomous model under some practical constraints in computerized classification testing (CCT), such as methods to control item exposure rate, and to study the effects of other variables, including item information algorithms, test difficulties, item…

The purposes of this study were to: (1) extend the sequential probability ratio testing (SPRT) procedure to polytomous item response theory (IRT) models in computerized classification testing (CCT); (2) compare polytomous items with dichotomous items using the SPRT procedure for their accuracy and efficiency; (3) study a direct approach in…

National Assessment of Educational Progress writing sample essays from 1988 and 1990 (495 and 599 essays) were subjected to computerized grading and human ratings. Cross-validation suggests that computer scoring is superior to a two-judge panel, a finding encouraging for large programs of essay evaluation. (SLD)

Large-scale testing is dominated by the multiple-choice question format. Widespread use of the format is due, in part, to the ease with which multiple-choice items can be scored automatically. This paper examines automatic scoring procedures for an alternative item type: figural response. Figural response items call for the completion or…

This paper reviews the salient characteristics of the current inventory of large-scale achievement tests, analyzes two recent proposals for creating a national testing program, and describes a new approach to statewide and nationwide testing that RAND is examining. Section 2 discusses the criteria typically used for evaluating large-scale testing…

This Monte Carlo study examined the effect of complex sampling of items on the measurement of differential item functioning (DIF) using the Mantel-Haenszel procedure. Data were generated using a three-parameter logistic item response theory model according to the balanced incomplete block (BIB) design used in the National Assessment of Educational…

Several developments growing out of the National Assessment of Educational Progress (NAEP) are reviewed, with a discussion of their extension and application to other projects. These developments include: (1) complex matrix item sampling designs; (2) performance-based items in large-scale assessments; (3) vertical scaling; and (4) innovative…

Indicators of current issues and concerns in the assessment of science teaching and learning outcomes are discussed. The greatest public attention to science test scores is devoted to scores from assessments that are external to schools, including national assessments such as the National Assessment of Educational Progress (NAEP), international…