Evaluators' experiences with being pressured by a key stakeholder to misrepresent findings were investigated in an anonymous survey of a random sample of over 2500 American Evaluation Association members; a response rate of 37% was obtained. Overall, 42% of the respondents had encountered misrepresentation pressure, with 70% of this subgroup…

Whether foreign language writing is rated using analytic rating scales or holistically, the organization of ideas is invariably one of the aspects assessed. However, it is unclear what raters are sensitive to when rating writing for organization. Which do they value more highly, the physical aspects of organization, such as paragraphing and the…

Previous Language Testing research has largely reported that although many raters' characteristics affect their evaluations of language assessments (Reed & Cohen, 2001), being a native speaker or non-native speaker rater does not significantly affect final ratings (Kim, 2009). In Second Language Acquisition, some researchers conclude that…

An essential goal of evaluation is to foster learning. Across the medical education spectrum, evaluation of clinical performance is dominated by subjective feedback to learners based on observation by expert supervisors. Research in non-medical settings has suggested that participants' perceptions of evaluation processes exert considerable…

Program and/or course evaluation is a process in which different types of data are collected systematically in order to study the virtues and weaknesses of a language instruction program. Program evaluation is, in fact, one of the essential aspects of any curriculum. It is a kind of quality control in which various aspects of an instructional…

Training is critical for the growth of the evaluation profession as more qualified and competent individuals will be required to meet the increased demand for evaluation. This paper deals with the training needs of professionals and is based on a part of a study on the status and challenges of educational program evaluation in the Asia-Pacific…

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff,…

Objective: The aim of this pilot project was to explore the extent to which judgments made by psychiatrist examiners accord with those of patients in postgraduate clinical examinations, so as to inform further consideration of the role of patients in such assessments. Method: Senior psychiatrist examiners (N=8) and patients (N=30) rated 16 aspects…

The examination of cross-classified category data is common in evaluation and research, with Karl Pearson's family of chi-square tests representing one of the most utilized statistical analyses for answering questions about the association or difference between categorical variables. Unfortunately, these tests are also among the more commonly…

Research in second language speech has often focused on listeners' accent judgment and factors that affect their perception. However, the topic of listeners' application of specific sound categories in their own perceptual judgments has not been widely investigated. The current study explored how listeners from diverse language backgrounds weighed…

This report focuses on the progress made and challenges faced by the nine states in achieving the first three objectives for which they were held accountable for use of their Race to the Top Early Learning Challenge (RTTELC) funds. It discusses discrete findings related to these objectives in relevant chapters of the report: Tiered Quality Rating…

This technical assistance paper provides guidance to individuals with test administration responsibilities in adult education programs. These policies apply to the approved assessments that programs may use to report educational gains in compliance with the National Reporting System (NRS) requirements. The NRS is the accountability system for the…

Developmental evaluation (DE), essentially conceptualized by Patton over the past 30 years, is a promising evaluative approach intended to support social innovation and the deployment of complex interventions. Its use is often justified by the complex nature of the interventions being evaluated and the need to produce useful results in real time.…

This study investigates how a group of 30 multilingual academics, all users of English as an additional language (EAL) working at a private university in Oman, acquired discourse community membership in their disciplines through publishing in English, and the strategies they use to sustain the level of literacy needed to disseminate their research…

In this paper we share our reflections, as evaluators, on an evaluation where we encountered Excessive Evaluation Anxiety (XEA). The signs of XEA which we discerned were particularly evident amongst the program head and staff who were part of a new training program. We present our insights on the evaluation process and its difficulties, as well as…