This article provides results from a study of the hidden processes of consensus formation that precede and make possible official judgments and decisions of the Swedish Schools Inspectorate (SI). The research question for the study was: How is knowledge negotiated on the back stage of school inspection and presented on the front stage? The article…

Using generalizability (G-) theory and rater interviews as both quantitative and qualitative approaches, this study examined the impact of scoring methods (i.e., holistic versus analytic scoring) on the scoring variability and reliability of an EFL institutional writing assessment at a Turkish university. Ten raters were invited to rate 36…

Assessors' scores in performance assessments are known to be highly variable. Attempted improvements through training or rating format have achieved minimal gains. The mechanisms that contribute to variability in assessors' scoring remain unclear. This study investigated these mechanisms. We used a qualitative approach to study…

Research in second language speech has often focused on listeners' accent judgment and factors that affect their perception. However, the topic of listeners' application of specific sound categories in their own perceptual judgments has not been widely investigated. The current study explored how listeners from diverse language backgrounds weighed…

Background: Turkey is a fast-developing country facing a lot of different problems. One of the problems is inadequate education. A large project was started to improve Turkish elementary education. The project includes many different purposes from physical refurbishment of school buildings, to increasing technology use in education. The author of…

This qualitative study was designed to examine how homeschooling parents in Pennsylvania make the determination to engage with public school districts to accommodate the special education needs (SEN) of their children. This phenomenological study used direct interviews with 30 Pennsylvania families who are homeschooling children with SEN. Data…

Think-aloud protocols (TAPs) are frequently used in research on essay rating processes. However, there are very few empirical studies of the completeness of TAP data and the effects of this technique on rater performance (i.e., rating processes and outcomes). This study aims to start to address this research gap. As part of a larger study on rater…

This chapter explores how the structure of large complex organizations such as Cooperative Extension affects their ability to support internal evaluation of their programs and activities. Following a literature review of organizational structure and its relation to internal evaluation capacity, the chapter presents the results of interviews with…

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site…

Second language (L2) oral performance assessment always involves raters' subjective judgments and is thus subject to rater variability. The variability due to rater characteristics has important consequential impacts on decision-making processes, particularly in high-stakes testing situations (Bachman, Lynch, & Mason, 1995; A. Brown, 1995;…

An interview with the Illinois State director of research and evaluation, covers issues related to commissioning and using evaluators. Pancrazio's position is with the State Board of Education. Focus is on the preparation of research reports to facilitate decision making and policy making. (TJH)

This paper presents a case study of a stakeholder-collaborative approach to the utilization of evaluation. The roles of participants were extended to incorporate: the design, administration, and interpretation of surveys and interviews; extensive data-gathering; and development of major portions of the final report. (Author/LMO)

During the past 3 years, the Oral History Project Study Team, which comprises Jean King, Mel Mark, and Robin Miller, has conducted interviews with individuals who have made signal contributions to the program evaluation field. Their goal was to capture the professional evolution of those who have contributed to the way evaluation in the United…

An interview with Michael Scriven describes his thoughts and contributions to educational evaluation, the evolution of his career, and his current concerns and projects. (MH)

An internal evaluation was carried out on the governing procedures of the Center for Curriculum and Instruction at the University of Nebraska-Lincoln during the spring semester, 1985-86. The evaluation focused on the major governance committee within the Center (the Center Executive Committee) and its relationship with the two other forms of…