In recent years, educators, administrators, policymakers, and measurement experts have called for assessments that support educators in making better instructional decisions. One promising approach to measurement to support instructional decision-making is diagnostic classification models (DCMs). DCMs are flexible psychometric models that…

Large language models are increasingly used in educational and psychological measurement activities. Their rapidly evolving sophistication and ability to detect language semantics make them viable tools to supplement subject matter experts and their reviews of large amounts of text statements, such as educational content standards. This paper…

In this digital ITEMS module, Dr. Stella Kim provides an overview of multidimensional item response theory (MIRT) equating. Traditional unidimensional item response theory (IRT) equating methods impose the sometimes untenable restriction on data that only a single ability is assessed. This module discusses potential sources of multidimensionality…

The Cross-Classified Mixed Effects Model (CCMEM) has been demonstrated to be a flexible framework for evaluating reliability by measurement specialists. Reliability can be estimated based on the variance components of the test scores. Built upon their accomplishment, this study extends the CCMEM to be used for evaluating validity evidence.…

In this ITEMS module, we introduce the generalized deterministic inputs, noisy "and" gate (G-DINA) model, which is a general framework for specifying, estimating, and evaluating a wide variety of cognitive diagnosis models. The module contains a nontechnical introduction to diagnostic measurement, an introductory overview of the G-DINA…

Student learning outcomes assessment has been increasingly used in U.S. higher education institutions over the last 10 years, partly fueled by the recommendation from the Spellings Commission that institutions need to demonstrate more direct evidence of student learning. To respond to the Commission's call, various accountability initiatives have…

The purpose of this ITEMS module is to provide an introduction to differential item functioning (DIF) analysis using mixture item response models. The mixture item response models for DIF analysis involve comparing item profiles across latent groups, instead of manifest groups. First, an overview of DIF analysis based on latent groups, called…

Mokken scale analysis (MSA) is a probabilistic-nonparametric approach to item response theory (IRT) that can be used to evaluate fundamental measurement properties with less strict assumptions than parametric IRT models. This instructional module provides an introduction to MSA as a probabilistic-nonparametric framework in which to explore…

Drawing valid inferences from item response theory (IRT) models is contingent upon a good fit of the data to the model. Violations of model-data fit have numerous consequences, limiting the usefulness and applicability of the model. This instructional module provides an overview of methods used for evaluating the fit of IRT models. Upon completing…

Explores how a program of strong construct validation could be applied to the assessment of the construct of test anxiety, paying special attention to substantive, structural, and external aspects of construct validation. A framework is proposed to pull together various statistical methods used in construct validation research into an organized…

The need for an independent mechanism that regulates, or audits, the testing enterprise is discussed along with a critique of current mechanisms for challenging a high-stakes test or its use and the need for independent auditing of the commercial test industry. Models for an auditing mechanism are reviewed. (SLD)

In this paper I describe and illustrate the Roussos-Stout (1996) multidimensionality-based DIF analysis paradigm, with emphasis on its implication for the selection of a matching and studied subtest for DIF analyses. Standard DIF practice encourages an exploratory search for matching subtest items based on purely statistical criteria, such as a…

If curriculum is to be the basis for assessment reform, assessment specialists must model the process for producing valid assessment products. Validity criteria should guide any model for the assessment development process. However, curriculum-based assessment systems should not be confused with standards-driven assessment systems. (SLD)

A more fragmented approach to assessment of global ability concepts than is generally advocated is suggested, based on the assumption that decomposing a complex ability into cognitive components and tracking performance across multiple measures will yield valid and instructionally useful information. Specifications are suggested for designing…

This article examines how psychometric models based on certain assumptions have come to be used counterproductively by many practitioners in ways that limit the kinds of conclusions that can be made. The general problem of the context's influence on performance is discussed, and some implications are drawn. (SLD)