This guide will help researchers design and implement impact studies in education so that the findings are more generalizable to the study's target population. Guidance is provided on key steps that researchers can take, including defining the target population, selecting a sample of schools--and replacement schools, when needed--managing school…

Researchers have explored a variety of topics related to identifying and distinguishing among specific types of rater effects, as well as the implications of different types of incomplete data collection designs for rater-mediated assessments. In this study, we used simulated data to examine the sensitivity of latent trait model indicators of…

This case study describes the process I engaged in to plan for and implement a design-based research project in an effort to explore what critical language study looked like in practice as well as how teachers and students responded to this instructional approach. In this case study, I focus, specifically, on the choice I made to take on the role…

The 4-H youth development organization is a complex public--private partnership between the U.S. Department of Agriculture's National Institute of Food and Agriculture, the nation's Cooperative Extension system and National 4-H Council, a private, nonprofit partner. The current article is focused on a partnership approach to the…

This article presents a content analysis of the 2013 School Leadership Program (SLP) grants. SLP projects provide a unique opportunity for participants in the field to explore innovative leadership preparation and development and their impact on program participants, schools, school districts, and students. The article begins with an overview of…

This research gathered data from a sample of university students with disabilities (and a control group without disabilities) from two university campuses (one urban and the other suburban). The project examined the current work and school experiences, plans upon graduation, and understanding of major legislation designed to protect against…

This article reflects on a university human research ethics committee's unease regarding a feminist visual pilot study within the field of education. The small exploratory study proposed to explore a migrant mother's production of her son's identity through her family photograph collection. The committee requested substantial…

The aim of the study was to investigate how raters come to their decisions when judging spoken vocabulary. Segmental rating was introduced to quantify raters' decision-making process. It is hoped that this simulated study brings fresh insight to future methodological considerations with spoken data. Twenty trainee raters assessed five Chinese…

Call it innovation, creativity, imagination, cutting edge, paradigm shift, or any other term for new information, an assessment of innovation may now impact the final decision on awarding grants to investigators. What exactly is innovation and how does the reviewer perceive innovation in the research approach? Procedures, the approach, and…

The goal of this chapter is to recommend quality criteria to guide evaluators' selections of sampling designs when mixing approaches. First, we contextualize our discussion of quality criteria and sampling designs by discussing the concept of interpretive consistency and how it impacts sampling decisions. Embedded in this discussion are…

Legitimate knowledge claims about causation have been a central concern among evaluators and applied researchers for several decades and often have been the subject of heated debates. In recent years these debates have resurfaced with a renewed intensity, due in part to the priority currently being given to randomized experiments by many funders…

The continuously growing demand for program results has produced an increased need for evaluation capacity building (ECB). The "Integrative ECB Model" was developed to integrate concepts from existing ECB theory literature and to structure a synthesis of the empirical ECB literature. The study used a broad-based research synthesis method with…

This study addresses the central question "How do evaluators' background characteristics relate to their evaluation design choices?" Evaluators were provided with a fictitious description of a school-based program and asked to design an evaluation of that program. Relevant background characteristics such as level of experience,…

Scientifically based research used to inform evidence based school reform efforts has been required by the federal government in order to receive grant funding since the reenactment of No Child Left Behind (2002). Educational evaluators are thus faced with the challenge to use rigorous research designs to establish causal relationships. However,…

This essay explores the contributions of "Evaluation Practice" and the "American Journal of Evaluation" to our understanding of evaluation ethics. The review is organized according to the five Guiding Principles for Evaluators (Systematic Inquiry, Competence, Integrity/Honesty, Respect for People, and Responsibilities for General and Public…