This paper describes, and lays out an argument for, the use of a procedure to help groups of reviewers to judge the quality of prior research reports. It argues why such a procedure is needed, and how other existing approaches are only relevant to some kinds of research, meaning that a review or synthesis cannot successfully combine quality…

Several AI-aided screening tools have emerged to tackle the ever-expanding body of literature. These tools employ active learning, where algorithms sort abstracts based on human feedback. However, researchers using these tools face a crucial dilemma: When should they stop screening without knowing the proportion of relevant studies? Although…

Researchers' notions of research quality depend on their field of research. Previous studies have shown that field-specific assessment criteria exist but could explain neither why these specific criteria and not others exist, nor how criteria are used in specific assessment situations. To give initial answers to these questions, formal assessment…

Previous research shows a discrepancy between different teachers' assessment of student theses. This might be an even larger problem in the context of teacher education, since teacher trainers originate from different disciplines. This study aims to investigate how different assessors prioritise between criteria for assessment. Criteria were…

The prevalence of mental health and emotional well-being difficulties in children is increasing, and schools play a key role in addressing this. Whole-school approaches have been suggested as an effective way of supporting children's mental health and well-being; however, there appears to be no consistent approach to their evaluation, and…

This article provides insight into how an adequate approach to selecting methods can establish credible and actionable evidence. The authors offer strategies to effectively support Extension professionals, including program developers and evaluators, in being more deliberate when selecting appropriate qualitative and quantitative methods. In…

Traditional classroom assessment practice often leaves students out of the conversation, exacerbating the unequal power distribution in the classroom. Viewing classrooms as autonomy-inhibiting is known to influence students' psychosocial wellbeing as well as their academic achievement. This is especially relevant in STEM fields where marginalized…

The focus of this paper is to share thinking about meta-level evaluation of action research (AR), and to introduce indicator domains for assessing and measuring inputs, outputs and outcomes. Meta-level and multi-site evaluation has been rare in AR beyond project implementation and participant satisfaction. The paper is the first of several…

Despite recent emphasis on the use of randomized control trials (RCTs) for evaluating education interventions, in most areas of education research, observational methods remain the dominant approach for assessing program effects. Over the last three decades, the within-study comparison (WSC) design has emerged as a method for evaluating the…

The purpose of this document is to describe the methods RAND used to analyze achievement for 23 personalized learning (PL) schools for the 2012-13 through 2013-14 academic years. This work was performed at the request of the Bill & Melinda Gates Foundation (BMGF), as part of a multi-year evaluation contract. The 23 schools were selected from a…

Since the mid 1970s, a series of international declarations that recognize the critical link between environmental sustainability and higher education have been endorsed and signed by universities around the world. While academic initiatives in sustainability are blossoming, higher education lacks a comprehensive evaluation framework that is…

Sound evaluation planning requires numerous decisions about how constructs in a program theory will be translated into measures and instruments that produce evaluation data. This article, the first in a dialogue exchange, examines how decisions about measurement are (and should be) made, especially in the context of small-scale local program…

This "Instructor's Corner" describes a step forward on the journey to write, review, and publish high-quality qualitative research manuscripts. This article examines two existing perspectives on generating high-quality qualitative manuscripts and then compares and contrasts the different elements of each. First, an overview of Rocco's (2010) eight…

This article draws on previously employed methods for conducting fidelity studies and applies them to an evaluation of an unprescribed intervention. We document the process of assessing the fidelity of implementation of the Math Recovery first-grade tutoring program, an unprescribed, diagnostic intervention. We describe how we drew on recent…

Research synthesis of evaluation findings is a multistep process. An investigator identifies a research question, acquires the relevant literature, codes findings from that literature, and analyzes the coded data to estimate the average treatment effect and its distribution in a population of interest. The process of estimating the average…