This research derived information functions and proposed new scalar information indices to examine the quality of multidimensional forced choice (MFC) items based on the RANK model. We also explored how GGUM-RANK information, latent trait recovery, and reliability varied across three MFC formats: pairs (two response alternatives), triplets (three…

As a global measure of precision, item response theory (IRT) estimated reliability is derived for four coefficients (Cronbach's a, Feldt-Raju, stratified a, and marginal reliability). Models with different underlying assumptions concerning test-part similarity are discussed. A detailed computational example is presented for the targeted…

In this article, procedures are described for estimating single-administration classification consistency and accuracy indices for complex assessments using item response theory (IRT). This IRT approach was applied to real test data comprising dichotomous and polytomous items. Several different IRT model combinations were considered. Comparisons…

This study explores a strategy for improving the stability of variance component estimates when only small samples are available, using a series of small, less complex generalizability (G) study designs as a surrogate for a single large design. (Author/BW)

Studied the effects of items, passages, contents, themes, and types of passages on the reliability and standard errors of measurement for complex reading comprehension tests using seven different generalizability theory models. Results suggest that passages and themes should be taken into account when evaluating the reliability of test scores for…

Four item response theory (IRT) models were compared using data from tests where multiple items were grouped into testlets focused on a common stimulus. In the bi-factor model each item was treated as a function of a primary trait plus a nuisance trait due to the testlet; in the testlet-effects model the slopes in the direction of the testlet…

Using several data sets, the relative performance of the beta binomial model and two more general strong true score models in estimating several indices of classification consistency is examined. It appears that the beta binomial model can provide inadequate fits to raw score distributions compared to more general models. (TJH)

A single conceptual and theoretical framework for sampling any configuration of data from one or more population matrices is presented, integrating past designs and discussing implications for more general designs. The theory is based upon a generalization of the generalized symmetric mean approach for single matrix samples. (Author/CTM)

An ignored aspect of standard setting, namely the possibility that Angoff or Nedelsky judges specify inconsistent probabilities (e.g., low probabilities for easy items but large probabilities for hard items) is explored. A latent trait method is proposed to estimate such misspecifications, and an index of consistency is defined. (Author/PN)

Explanations for an observed change in Rasch item parameters ("b" values) from consecutive administrations of a professional licensing examination were investigated. Analysis of covariance indicated that the change was not related to item position or type. It is hypothesized that the change is attributable to shifts in curriculum…

A procedure is described for estimating the reliability and conditional standard errors of measurement of scale scores incorporating the discrete transformation of raw scores to scale scores. The method is illustrated using a strong true score model, and practical applications are described. (SLD)

Racial differences in the reporting accuracy of parental status characteristics by White and Black high school seniors were investigated using Joreskog's general framework for simultaneous covariance structure analyses of multiple populations. Reliability estimates for Whites were significantly higher than for Blacks due to differences in true…