The purpose of this study was to develop and gather validity evidence for silent reading fluency passages. A number of passages were written following a traditional story grammar structure (character, setting, events) and placed on a computer for students to read silently. We describe in detail, the manner in which content-related evidence was…

The purpose of this study was to document the instrument development of maze measures for grades 3-8. Each maze passage contained twelve omitted words that students filled in by choosing the best-fit word from among the provided options. In this technical report, we describe the process of creating, reviewing, and pilot testing the maze measures.…

The purpose of this Item Response Theory study was to investigate how the expected reduction in item information, due to the collapsing of response categories in performance assessment data, was affected by varying testing conditions: item difficulty, item discrimination, inter-rater reliability, and direction of collapsing. The investigation used…

The main goal of this study was to assess how sensitive unidimensional parameter estimates derived from BILOG were when the unidimensionality assumption was violated and the underlying ability distribution was not multivariate normal. A multidimensional three-parameter logistic distribution that was a straightforward generalization of the…

Differential item functioning (DIF) assessment attempts to identify items or item types for which subpopulations of examinees exhibit performance differentials that are not consistent with the performance differentials typically seen for those subpopulations on collections of items that purport to measure a common construct. DIF assessment…

The Test of Spoken English (TSE) is an internationally administered instrument for assessing nonnative speakers' proficiency in speaking English. The research foundation of the TSE examination described in its manual refers to two sources of variation other than the achievement being measured: interrater reliability and internal consistency.…

Three papers describe the three stages of developing the National Teacher Examination (NTE) School Psychologist Specialty Area Test. The first stage is described in the paper entitled "Knowledge Areas Important to School Psychology." A survey of the membership of the National Association of School Psychologists helped determine knowledge…

The Mantel-Haenszel (MH) statistic for identifying differential item functioning (DIF) commonly conditions on the observed test score as a surrogate for conditioning on latent ability. When the comparison group distributions are not completely overlapping (i.e., are incongruent), the observed score represents different levels of latent ability…

The purpose of this paper is to clarify the seemingly discrepant views of test theorists and test developers about terminology related to the evaluation of test content. The origin and evolution of the concept of content validity are traced, and the concept is reformulated in a way that emphasizes the notion that content domain definition,…

Authentic and direct assessments of performances and products are conceptualized in terms of multiple distinctions having implications for validation. These include contrasts between performances and products, between assessment of performance per se and performance assessment of competence or other constructs, between structured and unstructured…

The relationship between the multiple-choice and free-response sections of the Computer Science and Chemistry tests of the College Board's Advanced Placement program was studied. Confirmatory factor analysis showed that the free-response sections measure the same underlying proficiency as the multiple-choice sections for the most part. However,…

This study describes the creation of measures of teachers' use of ability grouping in instruction using Rasch analysis. The dimensionality of the proposed construct was also investigated. Results of the Rasch analysis are compared to the results using composites to illustrate how the description of a construct can vary depending on the method used…

By focusing on "appropriateness" and "adequacy" of inference and action, unified validity may be misused in rejecting valid test outcomes. The notion of levels of validity is challenged, the necessity of assumption is argued, and experience is proposed as the basis of validity. "Consequential validity" is interpreted as an optional predictive…

Dichotomous item response theory (IRT) models can be viewed as families of stochastically ordered distributions of responses to test items. This paper explores several properties of such distributions. The focus is on the conditions under which stochastic order in families of conditional distributions is transferred to their inverse distributions,…

When running out of time on a multiple-choice test, some examinees are likely to respond rapidly to the remaining unanswered items in an attempt to get some items right by chance. Because these responses will tend to be incorrect, the presence of "rapid-guessing behavior" could cause these items to appear to be more difficult than they…