Researchers frequently evaluate rater judgments in performance assessments for evidence of differential rater functioning (DRF), which occurs when rater severity is systematically related to construct-irrelevant student characteristics after controlling for student achievement levels. However, researchers have observed that methods for detecting…

A quality management system is an essential and mandatory instrument for maritime higher education institutions mandated by the International Convention on Standards of Training, Certification, and Watchkeeping for Seafarers. To design an effective quality management system, it is strongly recommended to take the most comprehensive model of the…

Evaluations are practiced in political contexts, posing ethical dilemmas to evaluators. It is important, therefore, to prepare evaluation students for ethical decision-making in their future evaluative work. This study explores the use of scenario-based moral reasoning and ethical argumentation as an instructional strategy for teaching evaluation…

Evaluators play a central role in assessments of researchers' performance for reward, but the nature of their role and influence is not well understood. Ongoing reliance on evaluator judgement is typically justified as a need for referees in contests for reward, because quantitative performance measures alone can be subject to distortion. Yet, if…

A large data set of L1 psycholinguistic norms (Balota "et al." 2007) was used to assess spoken L2 English lexical proficiency in cross-sectional and longitudinal learner corpora. Behavioral norms included lexical decision and word naming latencies (i.e. reaction times) and accuracies for 40,481 English words. A frequency measure was…

There is emerging evidence that collocation use plays a primary role in determining various dimensions of L2 oral proficiency assessment and development. The current study presents the results of three experiments which examined the relationship between the degree of association in collocation use (operationalized as t scores and mutual…

This article provides results from a study of the hidden processes of consensus formation that precede and make possible official judgments and decisions of the Swedish Schools Inspectorate (SI). The research question for the study was: How is knowledge negotiated on the back stage of school inspection and presented on the front stage? The article…

This article describes an alternative approach to setting standards for performance assessments. The procedure was designed for use in low-budget, relatively low-stakes contexts where it is not possible to bring expert judges together. The procedure that allowed participant judges to work individually throughout the process was an effort to…

Educator and author Laurence E. Lynn recommends the case study as a method for training policy analysts to understand the cognitive style and uniqueness of the decision makers they serve. Analysts must be interdisciplinary, politically knowledgable, yet unbiased. Policy analysis, particularly by external institutions, is spreading to Congress. (CP)

Suggestions for broadening the utilization of evaluation research findings are discussed. Major program changes, based on evaluation, may not become obvious for several years; program actions may contradict findings and still be based on rational decision making processes; evaluation findings may have far-reaching effects which were not recognized…

The evaluator's role in planning/implementing an evaluation of a new program to train school administrators in performance appraisal skills is described. This case study of the evaluators' broadening roles uses E. Chelimsky's continuum (1987) to analyze political processes and decision making involved in the evaluation of a state-funded program.…

To examine the impact of program evaluation information upon decision making, case studies were conducted of five different programs: (1) a diagnostic teaching kindergarten program (K-LEAD), funded under the Elementary Secondary Education Act (ESEA) Title I; (2) Project MORE, an ESEA Title IV-C funded junior high school program dealing with school…

An analysis of the use of a survey on a state-level conservation tillage program was conducted to assess the effects of personal influences. Personal factors influenced timeliness, user's desire for and ownership of information, interaction among decision makers and the evaluator, methodological appropriateness and quality, and use planning. (TJH)

Evaluation is an act of persuasion directed to a specific audience concerning the solution of a problem. The process of evaluation is prescribed by the nature of knowledge--which is generally complex, always uncertain (in varying degrees), and not always propositional--and by the nature of logic, which is always selective. In the process of…