Recent times have seen the growth in the number and scope of interacting professional reform movements in science, centered on themes such as open research, research integrity, responsible research assessment, and responsible metrics. The responsible metrics movement identifies the growing influence of quantitative performance indicators as a…

Background: Rapid reviews involve a streamlined approach to knowledge synthesis. They are used to identify high-quality evidence for the purpose of informing decisions and initiatives, completed over relatively short timeframes, and have been found to reach conclusions that do not differ extensively from full systematic reviews. Although common in…

Since the George Floyd murder in 2020 and Black Lives Matter campaign, there has been increased interest among evaluators and PIs in Diversity, Equity and Inclusion (DEI) in their projects and evaluations. Our previous research showed a lack of clarity in understanding the distinction between the terms in practice and desire to engage with these…

The use of assessments that require rater judgment (i.e., rater-mediated assessments) has become increasingly popular in high-stakes language assessments worldwide. Using a systematic literature review, the purpose of this study is to identify and explore the dominant methods for evaluating rating quality within the context of research on…

Many studies of education interventions make claims about impacts on students' outcomes. Some studies have designs that enable readers to make causal inferences about the effects of an intervention but others have designs that do not permit these types of conclusions. To help policymakers, practitioners, and others make sense of study results, the…

The What Works Clearinghouse (WWC) evaluates research studies that look at the effectiveness of education programs, products, practices, and policies, which the WWC calls "interventions." Many studies of education interventions make claims about impacts on students' outcomes. Some studies have designs that enable readers to make causal…

New and novel uses of evaluation processes and findings are possible when a community of practice develops as evaluation stakeholders participate in multisite evaluations in multiple ways. Developing such communities takes advantage of what makes multisite evaluations special. This chapter uses the example of the Innovative Technology Experiences…

A team at the University of Minnesota conducted the Collaboratives for Excellence in Teacher Preparation (CETP) core evaluation between 1999 and 2004. The purpose of the CETP core evaluation was to achieve consensus among CETP project leaders and project evaluators on evaluation questions; to develop, pilot, and field test evaluation instruments…

In 1995, the National Science Foundation (NSF) contracted with principal investigator Iris Weiss and an evaluation team at Horizon Research, Inc. (HRI) to conduct a national evaluation of the Local Systemic Change for Teacher Enhancement program (LSC). HRI conducted the core evaluation under a $6.25 million contract with NSF. This program…

This paper addresses an important problem that may really be a pseudoproblem perpetuated by a current artificial or even political need to isolate researchers, evaluators, and theorists and put them in conceptual-methodological boxes. In this paper, we argue that scholars of all kinds should focus on the nouns (i.e., research, evaluation, theory)…

Scientifically based research used to inform evidence based school reform efforts has been required by the federal government in order to receive grant funding since the reenactment of No Child Left Behind (2002). Educational evaluators are thus faced with the challenge to use rigorous research designs to establish causal relationships. However,…

Instruments for evaluating the implementation of inquiry science in K-12 classrooms are necessary if evaluators and researchers are to know the extent to which programs are implemented as intended and the extent to which inquiry science teaching accounts for student learning. For evaluators and researchers to be confident about the quality of…

The field of evaluation faces a number of serious challenges in light of No Child Left Behind legislation, among them feasibility, resources, and blurring lines among research, evaluation, and assessment. At the same time, these challenges open the door for opportunities in evaluation. Now more than ever, the expertise of evaluators is needed and…

A district evaluator in a large Texas district examines new challenges arising since implementation of No Child Left Behind, relating to (1) navigating competing requirements in state and federal accountability systems; (2) evaluating effectiveness of sanctions districts are required to address; (3) using scientifically based research (SBR) to…

The impact of No Child Left Behind (NCLB) is usually understood in relation to schools and districts, but the legislation has also affected community-based organizations that operate school-linked programs. This case study of an after-school program in California demonstrates how educational accountability systems that emphasize students' academic…