The Interrater/Test Reliability System (ITRS) is described. The ITRS is a comprehensive computer tool used to address questions of interrater reliability that computes several different indices of interrater reliability and the generalizability coefficient over raters and topics. The system is available in IBM compatible or Macintosh format. (SLD)

To assess cross-observer agreement on personality profiles, an Index of Profile Agreement and an associated coefficient are proposed that take into account both the difference between the ratings and the extremes of their mean. Data from the Revised NEO Personality Inventory for 250 peer ratings/self-reports and 68 spouse ratings/self-reports…

Unlike a substantial part of reliability literature in the past, this article is concerned with weighted combinations of a given set of congeneric measures with uncorrelated errors. The relationship between maximal coefficient alpha and maximal reliability for such composites is initially dealt with, and it is shown that the former is a lower…

Evaluated external assessor ratings of the quality of the research team and the proposed research for proposals submitted to the Australian Research Council large grants program. Data from 313 evaluations show that the reliability of the assessments varied systematically with the number of assessors and the score considered. Discusses implication…

A linear combination of a set of measures is often sought as an overall score summarizing subject performance. The weights in this composite can be selected to maximize its reliability or to maximize its validity, and the optimal choice of weights is in general not the same for these two optimality criteria. We explore several relationships…

Numerous studies have shown increasing item reliabilities as an effect of the item position in personality scales. Traditionally, these context effects are analyzed based on item-total correlations. This approach neglects that trends in item reliabilities can be caused either by an increase in true score variance or by a decrease in error…

Two longitudinal studies involving 29 raters concerning the construct validity, temporal stability, and interrater reliability of the latent common factors underlying subjective assessments by human raters of personality traits in the stumptail macaque and the zebra finch illustrate the use of generalizability analysis to test prespecified…

The biasing effects of measurement error in path analysis models can be overcome by the use of latent variable models. In cases where path analysis is used in practice, it is often possible to use parcels as indicators of a latent variable. The purpose of the current study was to compare latent variable models in which parcels were used as…

The way in which trait and rater variance combine in multitrait-multirater (MTMR) performance appraisal data is explored. Implications of the confirmatory factor analytic model and the composite direct product (CDP) model for MTMR data are examined. Superior fit of the CDP model for four data sets is discussed. (SLD)

Cudeck and Browne (1983) proposed using cross-validation as a model selection technique in structural equation modeling. The purpose of this study is to examine the performance of eight cross-validation indices under conditions not yet examined in the relevant literature, such as nonnormality and cross-validation design. The performance of each…

The factorial validity of the Maslach Burnout Inventory (MBI) and the equivalence of factorial measurements and structure across groups were studied for 163 intermediate-grade teachers, 162 secondary school teachers, and 218 university teachers in Canada. Reasons why the MBI may not be appropriate for university educators are discussed. (SLD)