Methods of probability modeling to analyze rater agreement are described, emphasizing their basic similarities and viewing them as variants of a common methodology. Statistical techniques for analyzing agreement data are described to address questions such as how many opinions are required to make a medical diagnosis with necessary accuracy. Kappa…

This paper addresses the question: What is proper conduct for an internal evaluator? Selected ethical issues are examined, and strategies for responding effectively are proposed. Issues examined are: (1) pressure to downplay negative and emphasize positive findings; (2) reinforcement for nonthreatening evaluation activities; (3) greater interest…

Should an educational evaluator be trained as a content specialist, an evaluation specialist, or some combination of the two? A content specialist has demonstratable expertise in the field of study being evaluated, but not training in evaluation methods; the evaluation specialist has expertise in evaluation methods but may know little about the…

Five studies of holistic writing assessment procedures examined interactive relationships of the participants, processes, and products of writing assessment episodes. The first study examined practices in designing writing test prompts. The second study investigated the effects of variation in the specification of audience in a writing test prompt…

It has often been argued that all techniques of standard setting are arbitrary and likely to yield different results for different techniques or persons. This paper deals with a related but hitherto ignored aspect of standard setting, namely, the possibility that Angoff or Nedelsky judges misspecify the probabilities of the borderline student's…

This issue of FEEDBACK, a newsletter produced by the the Austin Independent School District Office of Research and Evaluation (ORE), illustrates the accuracy, validity, and fairness of ORE reports. The independence of the reports is explained. Internal and external quality controls are used to ensure reliability and accuracy of the reports.…

Evaluation practice within state education agencies (SEAs) is reviewed from a management consultant's perspective. The study is based upon a review of literature, discussions with the Northwest Regional Educational Laboratory Research on Evaluation Program, and visits to SEAs in California, Montana and Washington. The main findings of the study…

Given the concern for reliability in essay evaluation and the prospect of "error" variance in its absence, methods to promote interrater reliability in the evaluation of written compositions have been developed. These methods reduce variation in the value systems being applied by readers to texts, either by limiting the group of readers…

Occasionally novice evaluators of educational products do not recognize the full scope of their jobs and may feel that there isn't much to do because a product prototype won't be ready for some time. This brief dialogue attempts to illustrate that there are many things an evaluator can do before he or she sees a product--perhaps even too much.…

This exploratory study seeks to explain variation in conflict and ambiguity among a national sample of directors of school district research and evaluation units. The approach developed argues that variation in evaluation unit directors' role conflict and ambiguity is a function of both school district and evaluation unit characteristics, since…

Building on the finding that respondent belief in the efficacy of surveys is perhaps more important in determining feelings of burden than interview length, some possible approaches available to the Federal Evaluation Contractor (FEC) to reduce respondent burden are identified. These approaches aim to generate acceptance at the various levels…

The basic principles of the discrepancy evaluation model (DEM), developed by Malcolm Provus, are presented. The three concepts which are essential to DEM are defined: (1) the standard is a description of how something should be; (2) performance measures are used to find out the actual characteristics of the object being evaluated; and (3) the…

The purpose of this paper is to show that for an evaluation to be a successful change effort, the evaluator must not only "preach" (simply report facts) but also "meddle" (judge quality and recommend changes). It appears that if the evaluator does not take a leadership role in pushing for modification, these modifications will…

In American schools, the classroom teacher must act in two conflicting capacities: as a facilitator of learning and as an evaluator of his own facilitating activities. To avoid problems inherent in this, the evaluator role could be assigned elsewhere, as in the Boy Scouts' merit badge system. (SJL)

The evaluator as purveyor of truth may be seen as one more lobbyist competing for politicians' attention. This article explains how evaluators can present their results forcefully in the political arena. Available from: Evaluation, 501 South Park Avenue, Minneapolis, Minnesota 55415. (CP)