Performance assessments rely on the expert judgment of raters for the measurement of the quality of responses, and raters unavoidably introduce error in the scoring process. Defined as the tendency of a rater to assign higher or lower ratings, on average, than those assigned by other raters, even after accounting for differences in examinee…

In this study, we compared the effects of simulator-based virtual ultrasound scanning practice to classroom-based hands-on ultrasound scanning practice on participants' knowledge of FAST window quadrants and interpretation, and on participants' performance on live patient FAST exams. Twenty-five novice participants were randomly assigned to the…

Self-efficacy is thought to be important for resuscitation proficiency in that it influences the development of and access to the associated medical knowledge, procedural skills and crisis resource management (CRM) skills. Since performance assessment of CRM skills is challenging, self-efficacy is often used as a measure of competence in this…

This research was designed to obtain information on the validity and reliability of three new evaluation techniques: (1) The Simulated Patient Management Problem (PMP), a written simulation exercise; (2) The Simulated Diagnostic Interview (DI), an oral exercise; and (3) The Simulated Proposed Treatment Interview (PTI), another oral exercise. These…

Under a background condition of either recorded radio chatter or no radio chatter, the individual performances of two flights of mid-phase instrument student pilots were measured during a simulated instrument cross-country mission in the T-38 ground trainer. Operational constraints prevented the exercise of optimal experimental controls, thereby…

An experiment to assess the value of a shadowgraph simulator for driver training is described. Two groups of 36 pupils with no previous driving experience, and matched for age and sex, were taught to drive up to the standard of the DOE driving test. The pupils in one of the groups received their first three hours instruction on a shadowgraph…

The author views the primary purpose of the modern engineering laboratory as teaching the student how to experiment. An attempt is made to update the engineering laboratory and make it generally meaningful for all students, whether headed for graduate school or an industrial job. The intent is to broaden the experimental picture so that students…

To realize its full research potential, a need exists for the development of an automated objective pilot performance evaluation system for use in the Advanced Simulation in Undergraduate Pilot Training (ASUPT) facility. The present report documents the approach taken for the development of performance measures and also presents data collected…

This paper is concerned with measuring achievement levels of medical students. Precise tools are needed to assess the readiness of an individual to practice. The basic question then becomes, what can this candidate do, at a given time, under given circumstances. Given the definition of the circumstances, and the candidate's performance, the…

An IPPB (intermittent positive pressure breathing) inservice class was restructured to provide more practical skill experience and problem solving situations. Results of performance tests using a computer-monitored patient simulator indicated that previously used cognitive tests and subjective evaluations were not adequate for determining course…

Valid assessment of professional competence has proven to be an elusive goal. Objective tests, direct observation of performance, overall ratings of competence, and simulations have been tried and found wanting in one way or another. Objective test items are criticized as being unrealistic and therefore invalid. Direct observation tends to be very…

This article addresses performance testing based on criterion-referenced measurement. A brief description of performance objectives is presented, followed by a discussion of the different types of performance tests: pencil and paper, identification, simulation, and work sample. Checklists and other types of rating scales are identified. (CT)

A chart of the major action points in the course of developing tests for training evaluation and for qualifying trainees provides a framework for discussing the problems and practices of test development. Performance standards are the most common sources of trouble; the test developer must have unequivocal standards to work from. (AJ)

The 27 papers in this collection (26 of which were presented at the conference) are organized according to the eight topical sessions: (1) Improving Ability Measurement Using Different Item Formats, (2) Alternative Models for Adaptive Testing, (3) Psychological and Subgroup Effects, (4) Performance Testing by Interactive Simulation, (5)…

The literature on criterion referenced testing is full of discussions concerning whether classical measurement techniques are appropriate, whether variance is necessary, whether new indices of reliability are needed, and the like. What appears to be lacking, however, is a clear and simple discussion of why the problems occur. This paper suggests…