Competency-based assessment and an emphasis on obtaining higher-level outcomes that reflect physicians' ability to demonstrate their skills has created a need for more advanced assessment practices. Simulation-based assessments provide medical education planners with tools to better evaluate the 6 Accreditation Council for Graduate Medical…

This paper reports the results of an analysis of the relationship between scorer behaviors and score variability. Thirty-six essay scorers were interviewed and asked to perform a think-aloud task as they scored 24 essays. Each comment made by a scorer was coded according to its content focus (i.e. appearance, assignment, mechanics, communication,…

Several extensions to the Angoff method of standard setting are described that can accommodate characteristics of performance-based assessment. A study involving 12 panelists supported the effectiveness of the new approach but suggested that panelists preferred an approach that was at least partially conjunctive. (SLD)

A performance-standard setting procedure termed judgmental policy capturing (JPC) and its application are described. A study involving 12 panelists demonstrated the feasibility of the JPC method for setting performance standards for classroom teachers seeking certification from the National Board for Professional Teaching Standards. (SLD)

Discusses the role and consistency of judges' interpretations of teacher performance as part of an evaluative scheme for complex performance, with reference to the ideological framework of professional standards. The tension between assessment decisions and the recognition that assessment involves interpretation is explored. (SLD)

Differences in the Texas Teacher Appraisal System scores of teacher subgroups over 2 years were examined for 2,366 teachers for scores on individual domains, sums of scores of the 1st 4 domains, and overall summary performance scores, as well as appraiser differences. Implications for teacher evaluation are discussed. (SLD)

Whether adequate levels of interrater reliability could be obtained on a national, standardized examination using one examiner per observation was studied with 101 paired candidate observations on an examination for optometry. Results indicate that psychometrically sound judgments can be obtained with one examiner. (SLD)

The five articles of this theme issue focus on new developments in the evaluation of science education programs that have both substantive and theoretical implications for the field of evaluation. Past and current evaluation approaches are reviewed, and an overview is given of new assessments consistent with new curriculum developments. (SLD)

Praxis III is an assessment procedure that provides information for making instructional and licensing decisions about beginning teachers. The Praxis III Assessor's job is to interview the beginning teacher, observe the teacher in the classroom, score the teacher's performance on 19 criteria, and summarize the evidence for each score. The Assessor…

The scoring method that will be applied in the current 12th-grade science assessment project of the National Science Foundation and the Office of Educational Research and Assessment is described. The method, "graded mark-point" scoring, is modeled after procedures developed by P. Tamir for use in the performance exercises of the Israeli…

The effectiveness of alternative procedures to correct for rater leniency/stringency effects was studied when true scores were known. Ordinary least squares, weighted least squares, and imputation of the missing data consistently outperformed averaging the observed ratings; and the imputation technique was superior to the least squares methods.…

Performance rating systems frequently use multiple raters in order to improve the reliability of ratings. However, unless all candidates are rated by the same raters, some candidates will be at an unfair advantage or disadvantage solely because they were rated by more stringent or lenient raters. To obtain fair and accurate evaluations of…

The New Standards Project is a joint effort of the Learning Research and Development Center (LRDC) at the University of Pittsburgh (Pennsylvania) and the National Center on Education and the Economy toward creation of a national examination system based on performance assessments. This study explored the feasibility of comparing performance on…

The use of portfolio assessment as a method of evaluating the writing competence of elementary school students was studied. The study contained two components: (1) an empirical study of the utility and meaningfulness of using an analytic rubric developed for the evaluation of traditional writing samples to score student portfolios; and (2) a…

In this paper, performance assessments are cast within a sampling framework. A performance assessment score is viewed as a sample of student performance drawn from a complex universe defined by a combination of all possible tasks, occasions, raters, and measurement methods. Using generalizability theory, the authors present evidence bearing on the…