Diagnostic classification models (DCMs) are psychometric models that can be used to estimate the presence or absence of psychological traits, or proficiency on fine-grained skills. Critical to the use of any psychometric model in practice, including DCMs, is an evaluation of model fit. Traditionally, DCMs have been estimated with maximum…

Studied sources of measurement error variance in the Teacher Efficacy Scale (TES) (Gibson and Dembo, 1984). Used reliability generalization to characterize the typical score reliability for the TES and potential sources of measurement error variance across 43 studies. Also examined related instruments for measurement integrity. (SLD)

Given the potential value of reliability generalization (RG) studies in the development of cumulative psychometric knowledge, the purpose of this paper is to provide a tutorial on how to conduct such studies and to serve as a guide for researchers wishing to use this methodology. After some brief comments on classical test theory, the paper…

The question of least-squares weights versus equal weights has been a subject of great interest to researchers for over 60 years. Several researchers have compared the efficiency of equal weights and that of least-squares weights under different conditions. Recently, S. V. Paunonen and R. C. Gardner stressed that the necessary and sufficient…

Quality of performance data yielded by subjective judgment is of major concern to researchers in performance appraisal. However, some confusion exists in the analysis of quality on ratings obtained from different rating scale formats and from different raters. To clarify this confusion, a study was conducted to assess the quality of judgmental…

Generalizability theory (g-theory) was used, as an alternative to classical test theory, to evaluate measurement error in a behaviorally anchored role-play measure, highlighting the usefulness of this theory in instrument development. G-theory partitions an observed score into the universe score and error scores associated with separate sources of…

A primary goal of this study was to determine the extent to which item difficulty was related to item position and, if a significant relationship was found, to suggest adjustments to predicted item difficulty that reflect differences in item position. Item response data from the Medical College Admission Test (MCAT) were analyzed. A data set was…

Direct assessments of the accuracy with which raters can use a rating instrument are presented. This study demonstrated how surplus behavioral incidents scaled during the development of Behaviorally Anchored Rating Scales (BARS) can be used effectively in the evaluation of the newly developed scales. Construction of scenarios of hypothetical…

Measurement disturbances, such as guessing, startup, and plodding, often result in an examinee's ability being either over- or under-estimated by the maximum likelihood estimation employed in latent trait psychometric models. Several authors have suggested methods to lessen the impact of unexpected responses on the ability estimation process. This…

Previous studies of test item bias have investigated how different groups of examinees perform differently on a given set of items. These studies imply that examinees should be treated in a certain way because they are of a particular sex or race rather than as individuals in their own right, but it is unrealistic and unfair to assume such an…

This paper examines how various equating methods are affected by: (1) sampling error; (2) sample characteristics; and (3) characteristics of anchor test items. It reviews empirical studies that investigated the invariance of equating transformations, and it discusses empirical and simulation studies that focus on how the properties of anchor tests…

This paper addresses problems confronting law school teachers in grading law school exams and assigning letter grades. Using prototypical dialogue and scenarios, the paper examines mathematical and statistical issues that contribute to grading errors. Discussed in relation to real world data and the bar exam are: differential weighting, combining…