Performance assessments, scenario-based tasks, and other groups of items carry a risk of violating the local item independence assumption made by unidimensional item response theory (IRT) models. Previous studies have identified negative impacts of ignoring such violations, most notably inflated reliability estimates. Still, the influence of this…

An example application of portfolio assessment was developed and the model and estimates of reliability derived from the literature were then used to estimate the characteristics of an operational large-scale portfolio assessment program. Costs were estimated to put results in a realistic context. (SLD)

The application of generalizability theory to the reliability and error variance estimation for performance assessment scores is discussed. Decision makers concerned with performance assessment need to realize the restrictions that limit generalizability such as limitations that lead to reductions in the number of tasks possible, rater quality,…

Explores the relationship between measurement theory and practice, considering five broad categories of: (1) models, assumptions, and terminology; (2) reliability; (3) validity; (4) scaling; and (5) setting performance standards. It must be recognized that measurement is not an end in itself. (SLD)