Instruction and learning are increasingly based on competencies, causing a call for assessment methods to adequately determine competency acquisition. Because competency assessment is such a complex endeavor, one single assessment method seems not to be sufficient. This necessitates Competency Assessment Programs (CAPS) that combine different…

Studied the convergent and discriminant validity of multiple measures of perceived competence and obtained 12 separate multitrait-multimethod (MTMM) data sets for 1,228 children in grades 3 through 8. Examined the bias inherent in appraisals of children's competence and estimated manifest and latent variable mean differences between boys and girls…

Agreement between two independent reviews of each of 278 manuscripts was compared on an overall recommendation and on specific rating items. Agreement between reviewers on separate dimensions, the unweighted sum of the dimensions, and various weighted sums was no better than that for the overall recommendation itself. (SLD)

This study was undertaken to determine whether an evaluation model employing multiple methods of data collection and analysis might yield more useful information for improving lifelong learning courses than existing models. Major findings included: (1) learning satisfaction appears to be dependent on the instructional environment adults may be…

Quality assurance and evaluation research, like other fields of social research and its application, are confronted with a series of problems. In the present paper, I want first to give a list of such problems, although necessarily incomplete. It is then claimed that while there is no "perfect" solution to these problems, critical multiplism may…

The growth of educational teams in collaborative reform initiatives spurs a need for team assessments to assist in promoting functional team environments, productive team work, and member satisfaction. This study applied a multitrait-multimethod framework (D. Campbell and D. Fiske, 1959) to assess team functioning and team leadership. Four case…

Items from two North American instruments, designed to measure students' evaluation of teaching effectiveness, were administered to university students in different countries. Most of the items were judged by the students to be appropriate and important. Statistical analysis showed strong support for both the convergent and divergent validity of…

In evaluation, the whole is greater than the sum of the parts when quantitative and qualitative approaches and methods are combined. Some circumstances in which both approaches can be effectively combined are considered, and case studies of two evaluations using multiple methods are presented. (LMO)

This paper introduces the Website Acceptability Tiered Checklist (WATCH), a preliminary version of a multi-trait scale that could be used by instructors and students to assess the quality of websites chosen as source materials in students' research papers in a Humanities program. The scale includes bands for assessing: (i) the authority and…

This study is concerned with evaluating critical learning of 10-11 year-old students studying global citizenship through an online discussion environment. The evaluation is based on analysing the use of language as both a social reasoning tool and cognitive learning tool. Evaluation methods use: (1) content analysis of the online discussion…

A study was conducted to compare three methods of assessing course-level self-efficacy beliefs within a multitrait multimethod (MTMM) framework. The methods involved: (1) successfully performing a number of domain-related tasks; (2) obtaining specific letter grades in the course; and (3) successfully performing generic academic tasks in the…

This study explored the possibility of using the Multi-Trait Multi-Method (MTMM) approach of D. Campbell and D. Fiske (1959) to examine the educational ecology of classrooms. Using the MTMM approach, the study focuses on the extent to which multiple methods of inquiry (surveys, drawings, and videos) are valid indicators of classroom teaching and…

The way in which trait and rater variance combine in multitrait-multirater (MTMR) performance appraisal data is explored. Implications of the confirmatory factor analytic model and the composite direct product (CDP) model for MTMR data are examined. Superior fit of the CDP model for four data sets is discussed. (SLD)

An application of confirmatory factor analysis, using the LISREL program, to retrospective analysis of rating data common in health professions education was illustrated. A causal model was specified and tested. Although most measures clustered around latent traits, there was more method variance in the data set than trait variance. (Author/JAZ)

Correlational analysis, classical test theory, confirmatory factor analysis, and multilevel Rasch modeling were used to refine a measure of adolescents' exposure to violence (ETV). Interpersonal violence could be distinguished from other potentially traumatic events; it was also possible to distinguish three routes of exposure (victimization,…