Even when raters demonstrate agreement in the use of a measure, limited score variability or violation of often-ignored statistical assumptions can result in lower reliability estimates than intuitively expected. This article uses data drawn from two randomized controlled trials of schema therapy and cognitive behavioral therapy for the treatment…

This guide will help researchers design and implement impact studies in education so that the findings are more generalizable to the study's target population. Guidance is provided on key steps that researchers can take, including defining the target population, selecting a sample of schools--and replacement schools, when needed--managing school…

To begin statistical analysis, Bayesians quantify their confidence in modeling hypotheses with priors. A prior describes the probability of a certain modeling hypothesis apart from the data. Bayesians should be able to defend their choice of prior to a skeptical audience. Collaboration between evaluators and stakeholders could make their choices…

The What Works Clearinghouse (WWC) evaluates research studies that look at the effectiveness of education programs, products, practices, and policies, which the WWC calls "interventions." Many studies of education interventions make claims about impacts on students' outcomes. Some studies have designs that enable readers to make causal…

The author combines a literature review with a theoretical analysis of the interface between teacher researchers and Institutional Review Boards in higher education. Maintaining that teacher researchers are "creators of knowledge" (Castle, 2006, p. 2), the article explores the lack of fit between insider research with an emic design and the…

Written in a refreshing conversational style, this text thoroughly prepares students, program administrators, and new evaluators to conduct evaluations or to use them in their work. The book's question-driven focus and clear discussions about the importance of fostering evaluation use by building collaborative relationships with stakeholders set…

Researchers undertake a number of different research evaluation tasks, taking up a substantial part of their research time--estimated to about one work month per year for a professor. This paper addresses the various evaluator roles and tasks researchers take on, and the tensions they involve. How the research evaluator role may conflict with the…

Informal science education (ISE) experiences can provide powerful opportunities for people with disabilities to experience and learn about science. When designed to be inclusive, such experiences can lead people with disabilities to feel competent and empowered as science learners, generate excitement and enthusiasm for science, and be equitable…

Instruments for evaluating the implementation of inquiry science in K-12 classrooms are necessary if evaluators and researchers are to know the extent to which programs are implemented as intended and the extent to which inquiry science teaching accounts for student learning. For evaluators and researchers to be confident about the quality of…

Across many areas of psychology, concordance is commonly used to measure the (intragroup) agreement in ranking a number of items by a group of judges. Sometimes, however, the judges come from multiple groups, and in those situations, the interest is to measure the concordance between groups, under the assumption that there is some within-group…

The teaching and learning of Irish in primary school is both an important educational issue and central to the national language revitalization effort. The findings of Irish-language programme evaluations, therefore, are invariably scrutinized very closely by different sectors. This paper examines how the later stages of a major evaluation took…

Participatory research operates in a complex, dynamic social milieu and seeks to share the power inherent in knowledge generation with community partners. Institutional review boards (IRBs), however, typically operate from a framework that assumes asymmetrical power relations, hierarchically structured. This article argues that these differing…

All evaluators face the challenge of striving to adhere to the highest possible standards of ethical conduct. Translating the AEA's Guiding Principles and the Joint Committee's Program Evaluation Standards into everyday practice, however, can be a complex, uncertain, and frustrating endeavor. Moreover, acting in an ethical fashion can require…

Use of evaluation findings is a valued outcome for most evaluators. However, to optimize use, the findings need to be disseminated to potential users in formats that facilitate use of the information. This reflective case narrative uses a national evaluation of a multisite National Science Foundation (NSF) program as the setting for describing the…

This report is a product of an European Association for Quality Assurance in Higher Education (ENQA) Workshop "Assessing educational quality: Knowledge production and the role of experts" hosted by the Norwegian Agency for Quality Assurance in Education (NOKUT) in Oslo in February, 2008. The workshop gathered representatives from higher…