This article addresses an important problem that faces educators in assessing students' competence levels in learned tasks. Data from 165 students from Massachusetts and Minnesota in the United States are used to examine the validity of five assessment modes (multiple choice test, scenario, portfolio, self-assessment and supervisor rating) in…

Increasing emphasis on assessment and concern about assessment techniques have stirred interest in alternative assessment forms, for which evidence is needed about consequences, transfer of performance on specific assessment tests, and assessment fairness. Criteria concerning consequences, fairness, transfer-generalizability, cognitive complexity,…

Industrial and organizational psychologists for the Department of Defense have been working for the past 10 years to develop high fidelity measures of job performance for use in validating job selection procedures and standards. Information on developing and scoring performance exercises in the Job Performance Measurement (JPM) Project is…

The empirical validity of generalizability theory was investigated by applying two three-facet designs to data obtained in 1988 from administration of the Scientific Thinking and Research Skill Test (STRST). The decision validity of the STRST was also examined. Subjects were 125 fifth-grade and 125 sixth-grade students who were administered the…

An empirical investigation of methodological issues associated with evaluating treatment effect in single-subject research (SSR) designs is presented. This investigation: (1) conducted a generalizability (G) study to identify the sources of systematic and random measurement error (SRME); (2) used an analytic approach based on G theory to integrate…

The results are provided of an initial analysis of the reliability (generalizability) of the System for Teaching and Learning Assessment and Review (STAR) as a comprehensive measure of classroom teaching and learning for making teacher certification decisions. The STAR contains 140 indicators of teacher effectiveness and student learning, which…

Discusses issues relating to the reliability, validity, utility, and information value of unobtrusive testing of library reference services; provides suggestions for practical applications of these criteria; applies study findings to library decision making and planning; and identifies topics for further methodological refinement. (Author/CLB)

In this paper, performance assessments are cast within a sampling framework. A performance assessment score is viewed as a sample of student performance drawn from a complex universe defined by a combination of all possible tasks, occasions, raters, and measurement methods. Using generalizability theory, the authors present evidence bearing on the…

A team of university and public school mathematics educators designed performance-based mathematics assessment tasks designed to align with the Texas Assessment of Academic Skills for 93 students who had been identified as at-risk in mathematics. Scenarios were developed based on four contexts: (1) familiar activity; (2) social issue; (3)…

The teacher assessment that is the subject of this paper is an essentially informal activity. The teacher assesses the student by posing questions, observing activities, and evaluating work in a planned or ad hoc way. The information obtained may be partial or fragmented, but repeating such assessments over time will allow the buildup of a solid…

The purpose of this study is to investigate the content-specificity of communication skills. It investigates the reliability and dimensionality of standardized patient (SP) ratings of communication skills in an Objective Structured Clinical Examination(OSCE) for final year medical students. An OSCE consisting of seven standardized patient(SP)…

This paper assesses the credibility of a single total instrument score and various logical sub-scores derived from a series of summative judgments about the quality of teaching performance. The objectives were to compare the generalizability of alternative Teacher Performance Assessment Instrument (TPAI) scores, to compare the dependability of…

The 14 chapters of this book focus on the technical advances, advances in applied settings, and emerging topics in the testing field. Part 1 discusses methodological advances, Part 2 considers developments in applied settings, and Part 3 reviews emerging topics in the field of testing. Part 1 papers include: (1) "Advances in…

This report describes 12 studies dealing with the knowledge monitoring component of metacognition. It is assumed that knowledge monitoring is basic to other metacognitive activities, such as evaluating learning, selecting appropriate strategies, or planning, because distinguishing between what students know and do not know ought to be a…