This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

This study investigated the application of multidimensional item response theory (IRT) models to validate test structure and dimensionality. Multiple content areas or domains within a single subject often exist in large-scale achievement tests. Such areas or domains may cause multidimensionality or local item dependence, which both violate the…

In the attempt to identify or prevent unfair tests, both quantitative analyses and logical evaluation are often used. For the most part, fairness evaluation is a pragmatic attempt at determining whether procedural or substantive due process has been accorded to either a group of test takers or an individual. In both the individual and comparative…

In this study we examined variations of the nonequivalent groups equating design for tests containing both multiple-choice (MC) and constructed-response (CR) items to determine which design was most effective in producing equivalent scores across the two tests to be equated. Using data from a large-scale exam, this study investigated the use of…

The intent of large-scale assessment systems is to promote student achievement toward specific standards by holding schools accountable for the performance of all students. However, it is difficult to know whether large-scale assessment systems are having this intended effect as they are currently implemented. In this article, the authors examine…

This study was conducted to determine how items of the Stanford-Binet Intelligence Scale, Form L-M, had performed in the 1930's standardization sample in comparison with the 1972 standardization sample. (SB)

This invited special issue of "Early Education and Development" presents research related to the Early Development Instrument (EDI; Janus & Offord, 2007), a community tool to assess children's school readiness at a population level. In this editorial introduction, we first sketch out recent trends in school readiness research that call for a…

Statistical procedures employed in item analysis are based on the performance of the individual on a given test. If, however, one desires to assess the efficiency of some treatment, the researcher is interested in the performance of groups of students who received the treatment rather than in the performance of any one individual. In such cases…

The present study investigates whether the Early Development Instrument (Offord & Janus, 1999) measures school readiness similarly across different groups of children. We employ ordinal logistic regression to investigate differential item functioning, a method of examining measurement bias. For 40,000 children, our analysis compares groups…

This study investigated the feasibility of concurrently and randomly sampling examinees and items in order to estimate group achievement. Seven 32-item tests reflecting a 640-item universe of simple open sentences were used such that item selection (random, systematic) and assignment (random, systematic) of items (four, eight, sixteen) to forms…

This handbook treats a restricted set of statistical procedures for addressing some of the most prevalent technical issues that arise in domain-referenced testing. The procedures discussed here were chosen because they do not necessitate extensive computations. The five major sections of the paper cover: (1) item analysis procedures for using data…

The goal of this research project was to investigate the use of dynamic assessment to increase equity, fairness, and accuracy in the testing of abilities and achievement. Dynamic tests have been found to reveal developing expertise in underrepresented minorities around the world that is not revealed by conventional static tests. The…

Techniques for maximizing discrimination between groups in norm referenced measurement to reflect sensitivity to group differences inherent in the evaluation of multilevel educational systems are discussed. Data from the Beginning Teacher Evaluation Study (BTES) were used to examine relationships between reading and mathematics achievement and…

An alternative way of studying group differences is proposed based on ordering analysis using item hierarchies as a basis of comparison between two groups. Subjects were sets of twins in elementary school. Results showed that blacks and whites and males and females had similar item hierarchies for complex items. (Author/LMO)