This scope-review presents the milestones of how Hierarchical Rater Models (HRMs) become operable to used in automated essay scoring (AES) to improve instructional evaluation. Although essay evaluations--a useful instrument for evaluating higher-order cognitive abilities--have always depended on human raters, concerns regarding rater bias,…

Topic models are mathematical and statistical models used to analyze textual data. The objective of topic models is to gain information about the latent semantic space of a set of related textual data. The semantic space of a set of textual data contains the relationship between documents and words and how they are used. Topic models are becoming…

Scoring procedures for the constructed-response (CR) items in large-scale mixed-format educational assessments often involve checks for rater agreement or rater reliability. Although these analyses are important, researchers have documented rater effects that persist despite rater training and that are not always detected in rater agreement and…

An aspect of assessment which has received little attention compared with perennial concerns, such as standards or reliability, is the role of judgment in marking. This paper explores marking as an act of judgment, paying particular attention to the nature of judgment and the processes involved. It brings together studies which have explored…

Education Law Section 3012-c requires a new performance evaluation system for classroom teachers ("teachers") and building principals ("principals"). New York State will implement a statewide comprehensive evaluation system for school districts and boards of cooperative educational services (BOCES). The evaluation system is…

This paper reports the results of an analysis of the relationship between scorer behaviors and score variability. Thirty-six essay scorers were interviewed and asked to perform a think-aloud task as they scored 24 essays. Each comment made by a scorer was coded according to its content focus (i.e. appearance, assignment, mechanics, communication,…

Theory and implications for methods of assessing administrative performance in simulated exercises are presented. The rationale is given for the following: (1) developing simulated exercises; (2) measuring behaviors exhibited during the exercises; (3) training evaluators; (4) combining information across exercises; and (5) storing and retrieving…

The Department of Measurement, Statistics, and Evaluation (EDMS) at the University of Maryland is working to develop Master's degree programs that are oriented around developing assessment professionals for work in applied settings. Two fundamentally different sets of experiences are being developed: (1) assessment development, administration, and…

The semi-structured interview was investigated as a content-based assessment designed to take into account the complexity of teaching. A semi-structured interview licensing assessment for secondary mathematics teachers was developed and tested by the Connecticut State Department of Education. The scoring system converted the open-ended verbal…

The Montgomery County (Maryland) public school system has started using assessments other than multiple-choice tests because it is felt that this will provide school staff with better information about the success of the instructional program. One of the ways assessments can provide better information is by having teachers score student papers.…

Praxis III is an assessment procedure that provides information for making instructional and licensing decisions about beginning teachers. The Praxis III Assessor's job is to interview the beginning teacher, observe the teacher in the classroom, score the teacher's performance on 19 criteria, and summarize the evidence for each score. The Assessor…

The scoring method that will be applied in the current 12th-grade science assessment project of the National Science Foundation and the Office of Educational Research and Assessment is described. The method, "graded mark-point" scoring, is modeled after procedures developed by P. Tamir for use in the performance exercises of the Israeli…

Performance rating systems frequently use multiple raters in order to improve the reliability of ratings. However, unless all candidates are rated by the same raters, some candidates will be at an unfair advantage or disadvantage solely because they were rated by more stringent or lenient raters. To obtain fair and accurate evaluations of…

The New Standards Project is a joint effort of the Learning Research and Development Center (LRDC) at the University of Pittsburgh (Pennsylvania) and the National Center on Education and the Economy toward creation of a national examination system based on performance assessments. This study explored the feasibility of comparing performance on…

Across the field of educational assessment the case has been made for alternatives to the multiple-choice item type. Most of the alternative types of items require a subjective evaluation by a rater. The reliability of this subjective rating is a key component of these types of alternative items. In this paper, measures of reliability are…