Evaluated Angoff ratings from eight different occupational licensing examinations through generalizability theory to estimate the optimal number of raters. Results indicate that approximately 10 to 15 raters is an optimal target range. (SLD)

School-based health centers (SBHCs) provide physical and mental health services on school campuses to improve student health status, and thereby potentially facilitate student academic success. With a growing emphasis on school accountability and the simultaneous dwindling of resources at the federal, state, and local levels, SBHCs face increasing…

The purpose of this article is to discuss the problem of illusion in research, especially in program evaluation. Currently, there are multiple research traditions in program evaluation. Nevertheless, all research orientations are concerned with the role of researchers, for example, how they interpret their data and what evidence they offer for…

An approach to essay grading based on signal detection theory (SDT) is presented. SDT offers a basis for understanding rater behavior with respect to the scoring of construct responses, in that it provides a theory of psychological processes underlying the raters' behavior. The approach also provides measures of the precision of the raters and the…

Previous research suggests that college students bring gender biases to their evaluations of male and female instructors. Increasingly, college oral communication classes are utilizing peer evaluation of student speeches. This study of 330 student evaluators at a Midwestern university examined gender bias in student peer evaluations of public…

This scenario discusses a situation in which a proposal has been published by a consortium of foundations for an "external" evaluator to evaluate a replication at two new sites of a program they have been funding for many years. A proposal is received from Dr. Porto-Novo, who has been the external evaluator of the initial program for about 10…

This article describes the nature and utility of fiscal analysis in evaluating complex community interventions. Using New York University's evaluation of the Robert Wood Johnson Foundation's Urban Health Initiative as an example, the authors describe issues arising in defining and operationalizing constructs for fiscal analysis. The approachs…

This article aimed at finding out if participatory processes (group discussions, enactments, and others) do make a valuable contribution in communication-based project implementation/evaluation and the fight against HIV/AIDS. A case study backed by documentary analysis of evaluation reports and occasional insights from interviews stood as the main…

The emergence of theory-driven evaluation has drawn attention to the need for practical ways for evaluators to construct program theory to achieve the benefits of the approach. This article illustrates the use of concept mapping as a technique to explicate underlying program theory in the context of family support programs. In this example, 29…

This paper specifically reports on a comprehensive evaluation of competency based assessment within the mining industry in Queensland, Australia. However, it also addresses a broader problem associated with competency-based assessment already identified in the literature. The qualitative component of the study involved 83 people, nineteen mine…

Reviewer evaluations and recommendations for 853 manuscript submissions, over a span of 4 years, are analyzed for evidence of particularistic bias, reviewer agreement, and predictive validity for forecasting a published manuscript's citation impact. Attributes of the submitters, their affiliated institutions, and the reviewers have little…

The use of fieldwork experiences has been widely accepted by teachers of evaluation as a method of preparing future practitioners for challenges encountered in the field. However, several challenges must be addressed to make fieldwork experiences a useful tool in preparing students for their professional careers. The authors present a case study…

Evaluation standards can be useful not only as a framework for the design but as an assessment of particular evaluations. They can also serve as indicators for the developmental stage of evaluation practice in different areas or countries. Until recently, there has not been any large-scale and comprehensive empirical investigation of knowledge and…

The effective evaluation of educational projects is becoming increasingly important to funding agencies and to the individuals and organizations involved in the projects. This brief "how-to" guide provides an introductory description of the purpose and basic ideas of project evaluation, and uses authentic examples from four different astronomy and…

This article examines issues in developing valid and reliable direct observation of behavior. Suggestions are made to minimize the problems that threaten validity and reliability. The discussion is concluded by an examination of costs and benefits of direct observation and who pays them and who benefits from them. (Contains 5 tables and 2 figures.)