The authors of this article suggest three features of a common approach to impact measurement: harness operational data, use constructs with bounded flexibility, and develop a cadre of analysts who are skilled at interpreting reports. The analysts are the most crucial of these. Evaluators are well suited to step into these roles, but it will…

Two of the central issues in comparative judgement (CJ), which are perhaps underexplored compared to questions of the method's reliability and technical quality, are "what processes do judges use to make their decisions" and "what features do they focus on when making their decisions?" This article discusses both, in the…

This article presents a new tool called Critical Evaluation Capital (CEC) designed to address issues of equity and social justice in program evaluation. CEC is grounded in the tenants of critical race theory and inspired by Yosso's work on community cultural wealth which raises critical issues of positionality and access. CEC is a system for…

The aim of the present study was to examine Turkish teacher candidates' competency levels in writing different types of test items by utilizing Rasch analysis. In addition, the effect of the expertise of the raters scoring the items written by the teacher candidates was examined within the scope of the study. 84 Turkish teacher candidates…

This guide will help researchers design and implement impact studies in education so that the findings are more generalizable to the study's target population. Guidance is provided on key steps that researchers can take, including defining the target population, selecting a sample of schools--and replacement schools, when needed--managing school…

This study conducts a comprehensive quantitative evaluation of OpenAI's language model, ChatGPT 4, for grading Task 2 writing of the IELTS exam. The objective is to assess the alignment between ChatGPT's grading and that of official human raters. The analysis encompassed a multifaceted approach, including a comparison of means and reliability…

Education leaders play a fundamental role in shaping curricular expectations and students' access to books, write education professors Daniel Liou and Kelly Deits Cutler. But it's not easy to know how to respond when leaders are under intense political pressure to pull a book (or topic) from the proverbial shelves. Liou and Deits Cutler share five…

It is necessary to foster teachers' ability to design and implement instructions that encourage students to construct evidence and engage in argumentative discourse. The argument of this paper is that the continuous reflection and discussion as well as regular practice on instructional designs will promote pre-service science teachers' (PSSTs')…

Studies on automatic scoring systems in writing assessments have also evaluated the relationship between human and machine scores for the reliability of automated essay scoring systems. This study investigated the magnitudes of indices for inter-rater agreement and discrepancy, especially regarding human and machine scoring, in writing assessment.…

Expert intuition is increasingly considered to be a valid form of knowledge, and research has proven its effectiveness in judgment and decision making in various fields. Theorists seem to recognize the contributions of intuition within evaluative practice, but it has never been well-documented. This article presents a study on expert intuition,…

This article shares the findings of a qualitative case study examining the experiences of teacher leaders as they engaged as teacher evaluators alongside school principals. Data collection included observations and interviews with four teacher leaders and three school principals to answer these research questions: (1) What TDEM structures have…

In recent decades, a number of training environments have moved toward program approaches targeting the development of competencies. Because of their complexity, monitoring the development of those competencies is a considerable challenge. Our hypothesis is that a computerized system could help overcome this challenge if it is well accepted by its…

Instant feedback plays a vital role in promoting academic achievement and student success. In practice, however, delivering timely feedback to students can be challenging for instructors for a variety of reasons (e.g., limited teaching resources). In many cases, feedback arrives too late for learners to act on the advice and reinforce their…

The main objective of this Methods Showcase Article is to show how the technique of adaptive comparative judgment, coupled with a crowdsourcing approach, can offer practical solutions to reliability issues as well as to address the time and cost difficulties associated with a text-based approach to proficiency assessment in L2 research. We…

Teacher evaluation's relationship with instructional improvement is under-theorized in the literature. To address this gap, this paper uses a conceptual framework rooted in human, social, and material capital to analyze and synthesize findings from research conducted since 2009 on whether and under what conditions teacher evaluation stimulates…