As the use of diagnostic assessment systems transitions from research applications to large-scale assessments for accountability purposes, reliability methods that provide evidence at each level of reporting are needed. The purpose of this paper is to summarize one simulation-based method for estimating and reporting reliability for an…

In many large-scale assessment programs, achievement level descriptors (ALDs) provide a critical role in communicating what scores on the assessment mean and in interpreting what examinees know and are able to do based on their test performance. Based on their test performance, examinees are often classified into performance categories. The…

The assessment of differential item functioning (DIF) in polytomous items addresses between-group differences in measurement properties at the item level, but typically does not inform which score levels may be involved in the DIF effect. The framework of differential step functioning (DSF) addresses this issue by examining between-group…

Developed a new classification method for the logistic regression (LR) procedure for differential item functioning (DIF) based on methods used in the Simultaneous Item Bias test and conducted a simulation study to determine if the effect size measure affects the Type I error and power rates for the LR DIF procedure. Results show that inclusion of…

Results of 96 theoretical/empirical studies were reviewed to see if they support a taxonomy of 43 rules for writing multiple-choice test items. The taxonomy is the result of an analysis of 46 textbooks dealing with multiple-choice item writing. For nearly half of the rules, no research was found. (SLD)

Methodological procedures used in validating the theoretical structure of academic self-concept and validating associated measurement instruments are reviewed. Substantive findings from research related to modes of inquiry are summarized, and recommendations for future research are outlined. (TJH)

This article reports on the consistency of the achievement-level classifications (below basic, basic, proficient, and advanced) established in 1999 for the South Carolina Palmetto Achievement Challenge Tests (PACT; Huynh, Meyer, & Barton, 2000) of English language arts and mathematics. It also utilizes the PACT longitudinal data files of…

A procedure to detect differential item functioning (DIF) is introduced that is suitable for tests with a cutoff score. DIF is assessed on a limited closed interval of thetas in which a cutoff score falls. How this approach affects the identification of DIF items is demonstrated with real data sets. (SLD)

Application of person-fit statistics to cognitive diagnosis requires special efforts to detect normal and usual response patterns resulting from sources of misconception that are frequently observed among students. This study shows a solution for the problem by introducing an extension of a person-fit statistic developed by K. Tatsuoka (1985).…

A taxonomy of 43 rules for writing multiple-choice test items is presented, based on a consensus of 46 textbooks. These guidelines are presented as complete and authoritative, with solid consensus apparent for 33 of the rules. Four rules lack consensus, and 5 rules were cited fewer than 10 times. (SLD)

Several existing group-based statistics to detect improbable item score patterns are discussed, along with the cut scores proposed in the literature to classify an item score pattern as aberrant. A simulation study and an empirical study are used to compare the statistics and their use and to investigate the practical use of cut scores. (SLD)

Psychometricians continue to develop and evaluate methods for linking test scores, both horizontally and vertically. This article describes a social moderation process for articulating (i.e., linking) performance standards across grade levels for an operational state assessment program. The researchers used generated data to evaluate the likely…

Referral, placement, and retention decisions were analyzed using item response theory (IRT) to study whether classification decisions could be placed on the latent continuum of ability normally associated with test items and to study the existence of classification differential item functioning. Results with 352 kindergarten children demonstrate…

The consequences in terms of misclassifications of students that would occur by replacing the constructed-response portion of the Advanced Placement (AP) examinations in economics with more multiple-choice items were studied. The 1991 AP examinations in micro- and macroeconomics were used. Computer simulation found that a small but statistically…