Practitioners often struggle to assess reflective learning in the workplace because of difficulties conceptualizing reflection and its effects in the workplace. This article addresses this problem by offering a pragmatic approach to assessment that asks practitioners to specify why they are using reflection, what they are hoping to gain from it,…

The study, with 79 rehabilitation therapists evaluating 21 single-subject graphs, found that the low interrater agreement often associated with visual analysis of single-subject data may be improved by simple supplements (such as trend lines) to visually inspected charts. (Author/DB)

Case means and case failures in performance-based medical student evaluations were examined to evaluate the consistency of ratings made by two or more standardized patients (SPs) simulating the same case. Results demonstrate a need for caution in interpreting scores obtained from a case checklist completed by multiple SPs. (Author/MSE)

Methods of probability modeling to analyze rater agreement are described, emphasizing their basic similarities and viewing them as variants of a common methodology. Statistical techniques for analyzing agreement data are described to address questions such as how many opinions are required to make a medical diagnosis with necessary accuracy. Kappa…

Reports on a study of the use of musical scores and rating scales by 80 pianists who listened to 21 trials of solo piano music. Found that the use of musical scores and rating scales did not improve interrater reliability. Discovered that the subjects were less consistent when evaluating slow musical pieces than faster pieces. (CFR)

Two studies investigated interrater agreement among 10 clinical dental examiners who scored residual subgingival calculus after student scaling on 4,160 real and 92 manikin tooth surfaces. Interrater reliability was low. Results suggest a need in periodontics for effective examiner calibration methods and objective subgingival calculus detection…

A study compared student evaluations made by residency directors and deans in 2 medical schools, using 3 standard methods of ranking 20 students per school. Ordinal ranking showed substantial agreement for 15 of 16 residency directors. Two methods of clustering into fixed groups gave high agreement only for top students. (Author/MSE)

This study, involving five experienced and trained speech language pathologists, categorized the elicited imitations of five profoundly and five severely prelingually hearing-impaired subjects using the Phonetic Level Evaluation. Failure to obtain acceptably high levels of reliability suggests that this measure may not yet be an accurate and…

Asserts that the whole-language approach requires broader evaluation procedures most often associated with alternative assessment methods. Discusses issues related to alternative assessment, such as goal setting, portfolios, and interrater reliability, as they apply to social studies. (CFR)

A college faculty member who has graded Advanced Placement exam essays on U.S. government and politics, taken mostly by high school juniors and seniors, suggests that high school teachers and college faculty who assess the essays are not the best qualified persons to do so and that despite efforts to ensure consistency, the resulting scores are…

A faculty consultant to the Educational Testing Service for advanced placement (AP) test reading in U.S. government and politics responds to an article criticizing essay evaluation methods and criteria, finding in it a fundamental misunderstanding of the AP reading process and explaining why the essays are subject to less scrutiny for style,…

The Performance Review, Analysis, and Improvement System for Educators (PRAISE) is a formative evaluation instrument designed to improve the performance of school principals. The system appears to be reliable and valid and is flexible enough to accommodate the needs of a variety of schools. Sample items and categories of the instrument include…

The Bilingual Education Evaluation System was developed to help local bilingual education projects overcome evaluation obstacles and design sound, useful evaluations within the constraints of available funding and with responsiveness to current federal legislation and regulations. The evaluation system proposed involves a process component, an…