This scope-review presents the milestones of how Hierarchical Rater Models (HRMs) become operable to used in automated essay scoring (AES) to improve instructional evaluation. Although essay evaluations--a useful instrument for evaluating higher-order cognitive abilities--have always depended on human raters, concerns regarding rater bias,…

This article presents the performance of item response theory (IRT) models when double ratings are used as item scores over single ratings when rater effects are present. Study 1 examined the influence of the number of ratings on the accuracy of proficiency estimation in the generalized partial credit model (GPCM). Study 2 compared the accuracy of…

The Office of Planning, Research and Evaluation (OPRE), a unit within the Administration for Children and Families (ACF), is responsible for advising the Assistant Secretary for Children and Families on increasing the effectiveness and efficiency of programs to improve the economic and social well-being of children and families. In collaboration…

This handbook is part of a series prepared by the Center for the Study of Evaluation (CSE) for evaluation workshop participants. Activities are suggested for teams of trainees to practice the use of the five stage evaluation model. The five stages are needs assessment, program planning, implementation evaluation, progress and outcome evaluation.…

The handbook on progress evaluation was prepared for workshop participants. The Center for the Study of Evaluation (CSE) conducts a series of workshops for administrators and other evaluation team personnel. Specific exercises and activities are suggested using the five stage evaluation model. The phases are needs assessment, program planning,…

Using examples from an evaluation design of a bilingual preschool program, the authors have developed a paradigm resting on contextually based evaluation and involving implementation strategies and participant researchers. Multiple methods of gathering environment-specific data are used in the process. (MH)

This speech concerns the dimensions of context-oriented evaluation and begins with a discussion of the differences between measurement, research, statistical methodology, and evaluation. Decision-oriented evaluation is described as a process in which the evaluator has, as a major concern, the responsibility to determine likely decisions prior to…

The structure and foundations of educational evaluation activities are explained. Evaluation theories are divided into descriptive or empirical, and normative. Descriptive theory is defined as explaining phenomena, rather than being based on normative premises. It is referred to as educational evaluation analysis and is classified as a social…

Beginning in 1970, the Cincinnati Public Schools experienced a shift in emphasis from centralized evaluation to local school program development and evaluation. Three functional models currently operate to evaluate three programs: Title I, alternative integration, and disadvantaged students. Each is distinguished by the level of trust existing…

Following an overview of the need for evaluation training, the Analysis of Evaluation Training Materials Project is described and the coverage, methodology, and usability of materials available for training educational personnel in evaluation skills are considered. Problem areas in evaluation, identified by the National Institute of Education…

Program administrators for 64 Elementary Secondary Education Act (ESEA) programs in New York City were asked to respond to a questionnaire rating scale concerning the helpfulness and quantity of information that had been provided to them by the program evaluators. The rating scale categorized areas of administrative decision making in five…

This paper addresses the main issues related to concepts of school effectiveness and their implications, with a focus on the selection of school-performance indicators. School effectiveness has many definitions, which is in part due to the different demands made by various social groups. The boundaries of performance within which schools should…

Based upon the premise that school divisions in Manitoba, Canada, should evaluate and improve upon themselves, this evaluation model was developed. The participating personnel and the development of the evaluation model are described. The model has 11 parts: (1) needs assessment; (2) statement of objectives; (3) definition of objectives; (4)…

The Center for Interactive Technology, Applications, and Research at the College of Engineering of the University of South Florida (Tampa) has developed objective and descriptive evaluation models to assist in determining the educational potential of computer and video courseware. The computer-based courseware evaluation model and the video-based…