This technical report describes the results of analyses performed on data collected from 2013 to 2017, using the IDEA Feedback System for Administrators (FSA). The FSA is used to gather impressions from core constituents about an administrator's performance of relevant administrative roles, as well as her/his leadership style, interpersonal…

In this book the authors explain how to build, and over time improve, the elements of an observation system that equips all observers to identify and develop effective teaching. It is based on the collective knowledge of key partners in the Measures of Effective Teaching (MET) project--which carried out one of the largest-ever studies of classroom…

The IDEA Feedback for Administrators system provides feedback to academic administrators about their performance of relevant administrative responsibilities and their leadership style and interpersonal characteristics. The system is based on a model of reflective practice, which is consistent with The IDEA Center's longstanding approach to…

Case means and case failures in performance-based medical student evaluations were examined to evaluate the consistency of ratings made by two or more standardized patients (SPs) simulating the same case. Results demonstrate a need for caution in interpreting scores obtained from a case checklist completed by multiple SPs. (Author/MSE)

Interrater reliability in numerical ratings of clerkship performance (n=1,482 students) in five surgery programs was studied. Raters were classified as accurate or moderately or significantly stringent or lenient. Results indicate that increasing the proportion of accurate raters would substantially improve the precision of class rankings. (MSE)

A quantitative study modeled the inherently subjective admissions process for 592 graduate school candidates and 72 raters. Logistic regression models were well-fitting and parsimonious, allowing analysis of each stage of the process. Extended committee discussion/deliberation phases were of limited productivity when inter-rater agreement was…

Two studies investigated interrater agreement among 10 clinical dental examiners who scored residual subgingival calculus after student scaling on 4,160 real and 92 manikin tooth surfaces. Interrater reliability was low. Results suggest a need in periodontics for effective examiner calibration methods and objective subgingival calculus detection…

Medical school admission interview ratings for four entering classes (n=356 students) were compared with preadmission academic variables (admission test scores, undergraduate grades), student characteristics (age, gender, residence), and interviewer characteristics (gender, professional background, admission committee membership). Recommendations…

Asserts that the whole-language approach requires broader evaluation procedures most often associated with alternative assessment methods. Discusses issues related to alternative assessment, such as goal setting, portfolios, and interrater reliability, as they apply to social studies. (CFR)

Group process and rating techniques were used to create a scale for measuring severity and frequency of 24 problem behaviors in institutionalized mentally retarded persons. A substantial correlation was found between scores and staff-time allocations, indicating usefulness of the scale as a predictor of staffing requirements for behavior…

This packet contains a manual and a workbook for developing performance tests in vocational education. The manual gives an in-depth description of how to develop, score, and use performance tests. It includes the following sections: definitions of performance testing, steps in developing a performance test, selecting a performance development…