Evaluation designs seek to provide evidence of impact and collect formative data to understand and improve programs and inform decisions. Formative data is commonly used at the onset of developing a new program to modify and improve before full implementation. However, formative data is often collected throughout the life of the program to shape…

The quality of a research study application sends a distinct signal to the institutional review board (IRB) about the skills, capacities, preparation, communication, experience, and resources of its authors. However, efforts to research and define IRB application quality have been insufficient. Inattention to the quality of an IRB application is…

Various areas of educational research are interested in the transitions between different states--or events--in sequential data, with the goal of understanding the significance of these transitions; one notable example is affect dynamics, which aims to identify important transitions between affective states. Unfortunately, several works have…

Human Research Ethics Committee (HREC) reviewers often provide similar feedback across applications, which suggests that the problem lies in researcher awareness of key issues rather than novel, unsolvable challenges. If common problems can be addressed before lodgement by applicants referencing clear evidence-based supports (e.g., FAQs on common…

Panahi and Mohebbi review James Dean Brown's 50-years of research in language testing, curriculum development and research statistics with reference to an impressionistic framework for analysis containing two components with their subcomponents: Annotations (i.e., briefing and implications) and main concepts and themes (i.e., testing and teaching…

This research aims to make a methodological contribution to the mixed methods literature by investigating the value-added of empirical studies that are explicitly framed with the purpose of interrogating divergent findings. Case studies introduce the role philosophical paradigms can play in reconciliation process and reveal the potential to…

Many practitioner researchers strive to understand which assessment practices have the best impact on learning, but in authentic educational settings, it can be difficult to determine whether one intervention, for example the introduction of an online quiz to a course studied by diverse students, is responsible for the observed effect. This paper…

The purpose of this article is to clarify the discrepancy in the use of "queer" as a sexual identity classification in education survey research. This study extends the work completed by Dugan and Yurman (2011), who empirically demonstrated problems with treating LGB students as a homogenous population through collapsing all respondents…

This article discusses the nonuse, misuse, and proper use of pilot studies in experimental evaluation research. The authors first show that there is little theoretical, practical, or empirical guidance available to researchers who seek to incorporate pilot studies into experimental evaluation research designs. The authors then discuss how pilot…

Evaluation research conducted in agencies that sanction law violators is often challenging and due process may preclude evaluators from using experimental methods in traditional criminal justice agencies such as police, courts, and corrections. However, administrative agencies often deal with the same population but are not bound by due process…

Successful school reform is dependent on the quality of decisions made by educational leaders. In such decision making, educational leaders are charged with using sound research findings as the basis for choosing school reform initiatives. As part of the debate regarding the usability of various evaluative research designs in providing information…

The purpose of this study is to help ensure that strategies for differential item functioning (DIF) detection for students with disabilities are appropriate and lead to meaningful results. We surveyed existing DIF studies for students with disabilities and describe them in terms of study design, statistical approach, sample characteristics, and…

The Council for the Accreditation of Educator Preparation (CAEP) requires faculty from educator preparation programs to provide evidence of credential candidates' impact on K-12 student learning. However, there is a paucity of information on preparation programs' use of direct assessments of student learning to gauge credential candidate…

To date, evaluation research and policy analysis have focused mainly on average program impacts and paid little systematic attention to their variation. Recently, the growing number of multi-site randomized trials that are being planned and conducted make it increasingly feasible to study "cross-site" variation in impacts. Important…

The important but little understood problem that motivated this study was the lack of research on valid assessment methods to determine progress in higher-order learning in situations involving complex and ill-structured problems. Without a valid assessment method, little progress can occur in instructional design research with regard to designing…