Assessing the measurement model (MM) of self-report scales is crucial to obtain valid measurements of individuals' latent psychological constructs. This entails evaluating the number of measured constructs and determining which construct is measured by which item. Exploratory factor analysis (EFA) is the most-used method to evaluate these…

This article presents a comparative judgment approach for holistically scored constructed response tasks. In this approach, the grader rank orders (rather than rate) the quality of a small set of responses. A prior automated evaluation of responses guides both set formation and scaling of rankings. Sets are formed to have similar prior scores and…

Self-report surveys are widely used to measure adolescent risk behavior and academic adjustment, with results having an impact on national policy, assessment of school quality, and evaluation of school interventions. However, data obtained from self-reports can be distorted when adolescents intentionally provide inaccurate or careless responses.…

Many measurement biases affect student evaluations of instruction (SEIs). However, two have been relatively understudied: halo effects and ceiling/floor effects. This study examined these effects in two ways. To examine the halo effect, using a videotaped lecture, we manipulated specific teacher behaviors to be "good" or "bad"…

This study examined the cultural relevance of adult attachment within a Korean sample (N = 390) using Rasch rating scale modeling. The psychometric properties of scores from the Korean version of the Revised Experiences in Close Relationships, comprised of two subscales of Anxiety (self) and Avoidance (other), were assessed. Results obtained from…

This article addresses the problem of accounting overall multivariate chance-corrected interobserver agreement when targets have been rated by different sets of judges (not necessarily equal in number). The proposed approach builds on Janson and Olsson's multivariate generalization of Cohen's kappa but incorporates weighting for number of judges…

This article presents a formula for weighted kappa in terms of rater means, rater variances, and the rater covariance that is particularly helpful in emphasizing that weighted kappa is an absolute agreement measure in the sense that it is sensitive to differences in rater's marginal distributions. Specifically, rater mean differences will decrease…

In a study involving eight judges, analysis with the FACETS model provides evidence that judges grade differently, whether or not scores correlate well. This outcome suggests that adjustments for differences among judges should be made before student measures are estimated to produce reproducible decisions. (SLD)

A graphical method is introduced to help identify important properties of an ambivalence index. When five ambivalence indexes were compared in this way, only two were found satisfactory. Comparison of the same indexes with ratings of 445 college students of 26 attitude topics found relatively small empirical differences among the indexes. (SLD)

A new approach based on Rasch measurement theory is described for examining the quality of ratings from standard-setting judges. Ratings of nine judges for 213 items on a nursing examination show that judges vary in their views of the essential items for nursing certification, with statistically significant variability in the judged essentiality…

A method is proposed for manipulation of scale recalibration to evaluate techniques designed to detect change due to scale recalibration, including the Ideal Scale Approach of R. Zmud and A. Armenakis (1978). An illustration of an adaptation of the Ideal Scale Approach with 194 undergraduate students is included. (SLD)

The scale invariance of the Pediatric Evaluation of Disability Inventory was examined when 412 children were rated by parents on functioning at home and the same children were rated by rehabilitation personnel in an educational setting. Results indicated that parents and personnel can be trained to make similar judgments. (SLD)

The Beck Depression Inventory (BDI) is one of the most frequently used instruments in the study of depression both within and outside of the United States. Though developed primarily with European American clinical populations, the BDI has been applied in nonclinical and non-Western samples. To determine whether such a practice is warranted, the…

An effort was made to determine the factorial validity of a preliminary scale for student evaluation of group counseling. Results of this factor analysis and those of a detailed item analysis were used to develop a new form of the Scale for the Evaluation of Group Counseling Experiences. (Author/GK)

A method is proposed to combine a relative approach and an absolute approach to performance appraisal by combining graphic rating and ranking. Two studies involving 196 college undergraduates who rated their instructors illustrate the promise offered by the proposed Qualitative Ranking Scale. (SLD)