The accuracy of 210 college undergraduates' judgments of the relative learnability of original (Army training manuals) and rewritten (war or cultural histories) versions of 20 pairs of texts of known difficulty was studied. Students were 95 percent accurate in their judgments of learnability. Implications for college textbook selection are…

This article describes an ongoing survey that illustrates how evaluators, working with program managers, have effectively used client-based data to monitor performance in the Social Security Administration. The value of client-based data outweighs the limitations and problems collecting it. Customer perceptions are a critical barometer of quality.…

This study is an in-depth investigation of the NBPTS bias reduction training from the perspective of assessors. The research examined how successful the bias training was in guiding assessors to recognize their biases and to identify actions to be used to reduce the impact of bias. The study focused on portfolio video entries to allow for a…

Antidepressant medication is considered the current standard for severe depression, and cognitive therapy is the most widely investigated psychosocial treatment for depression. However, not all patients want to take medication, and cognitive therapy has not demonstrated consistent efficacy across trials. Moreover, dismantling designs have…

The primary purpose of this study was to investigate the impact of the level of observed performance on the feedback strategy selected by a performance evaluator. One hundred and twenty-three actual performance evaluators from 15 different organizations and 123 college students reviewed, in groups which ranged from 2 to 20, a job description for…

Although a necessary component of good program management and accountability, monitoring data do not usually provide answers to questions about whether, how, and why a specific program works. More in-depth, systematic inquiry is needed to do this. In this article, the authors focus on the evaluation of human immunodeficiency virus (HIV)-prevention…

The purpose of this concluding chapter has been to stimulate creative thinking about how to use cases for evaluation teaching and training. The preparation of professional evaluators presents special challenges. Imaginative new teaching and training resources are appearing, as evidenced by the special sections on evaluation teaching that now…

In 1999, the staff of the Joint Legislative Audit and Review Commission (JLARC) of the Virginia General Assembly (the state legislature) completed a review of the Virginia Board of Medicine that found problems with the process used to regulate physicians and recommended major changes to a system that had remained relatively unchanged for almost…

The Accrediting Council for Independent Colleges and Schools (ACICS or "Council") supports the concept that periodic evaluation entered into voluntarily by institutions and their peers enhances the quality of the educational process and demonstrates that self-regulation of a profession is superior to outside regulation. To help institutions and…

This report addresses the reliability of the American Council on the Teaching of Foreign Languages (ACTFL) Oral Proficiency Interview (OPI), not as a measure of speaking ability, but rather as practiced by testers trained by the ACTFL, such as by the Interagency Language Roundtable (ILR), in English as a Second Language (ESL), French, German,…

This paper presents the findings of a qualitative followup survey of a sample of programs listed in the 1986 "Directory of Evaluation Training Programs." Four programs listed in the directory are no longer considered evaluation training, and five, originally similar, are still going strong. A sampling strategy was used to compare the discontinued…

A four-level taxonomy is proposed to define the usefulness of rater written comments for supporting letters of recommendation. The taxonomy is used to classify comments on 220 rating forms by 25 raters from two surgery departments regarding performance by third-year medical students. Written comments were classified by the following taxonomy: (1)…

Program design can aid problem-solving or decision-making tasks faced by evaluators. It can be used to facilitate evaluation planning through the definition of program boundaries and the generation and focusing of evaluation questions. Further, it can provide evaluation information about theoretical, historical, and political contexts of a…

This paper examines whether individuals who train themselves to score a performance assessment will rate acceptably when compared to known standards. Research on the efficacy of rater self-training materials developed by the Center for Applied Linguistics for the Texas Oral Proficiency Test (TOPT) is examined. Rater self-materials are described…

Three examinations administered to medical students were analyzed to determine differences among severities of judges' assessments and among grading periods. The examinations included essay, clinical, and oral forms of the tests. Twelve judges graded the three essays for 32 examinees during a 4-day grading session, which was divided into eight…