Modern-day educators often share a sense they lack voice and agency with school administration. A classic example is curriculum development, where third-party designers develop uncontextualized curricula, and teachers then must implement the design even if inefficient and ineffective. What happens if this identical situation occurs at the program…

Innovation agencies, that is organizations with the primary focus of funding applied research and technological development, evaluate project proposals to select the most promising proposals for funding. At the moment, there is only little verified knowledge available on project evaluation and selection processes of innovation agencies. We want to…

Researchers have documented the impact of rater effects, or raters' tendencies to give different ratings than would be expected given examinee achievement levels, in performance assessments. However, the degree to which rater effects influence person fit, or the reasonableness of test-takers' achievement estimates given their response patterns,…

This article shares lessons from the field with program evaluations in the English-Speaking Commonwealth Caribbean (ESCC) region. The research highlighted that the challenges faced by evaluators working in the ESCC are quite similar to those experienced by evaluators elsewhere. However, the findings note the impact of the region's colonial past…

Purpose: Multiple-measure principal evaluation systems have become commonplace in the past decade, but we do not know how principals perceive their evaluations under these regimes. This study analyzes how principals perceive evaluation in a state that was an early adopter of such a system. It describes how attitudes are explained by individual and…

Scoring procedures for the constructed-response (CR) items in large-scale mixed-format educational assessments often involve checks for rater agreement or rater reliability. Although these analyses are important, researchers have documented rater effects that persist despite rater training and that are not always detected in rater agreement and…

Studies of interaction in speaking assessment have highlighted problems regarding the unequal distribution of interaction patterns in different task types. However, little attempt has been made to include both verbal and nonverbal interaction features elicited in the tasks. Therefore, this study examined the elicitation of verbal and nonverbal…

Purpose: Multiple-measure principal evaluation systems have become commonplace in the last decade, but we do not know how principals perceive their evaluations under these regimes. This study analyzes how principals perceive evaluation in a state that was an early adopter of such a system. It describes how attitudes are explained by individual and…

Evaluators are becoming increasingly aware that, to provide maximum benefit to the programs they evaluate, they must address the systems within which the programs work. However, evaluation practice is often influenced by beliefs and assumptions that are rooted in an understanding of these systems as stable and predictable. We identify four traps…

When practitioners use modern measurement models to evaluate rating quality, they commonly examine rater fit statistics that summarize how well each rater's ratings fit the expectations of the measurement model. Essentially, this approach involves examining the unexpected ratings that each misfitting rater assigned (i.e., carrying out analyses of…

In this practice paper I examine the evaluation of one project and ask whether or not the evaluation failed. I suggest that because I was unaware of the larger connections between the grant agency and the grantee I made tactical errors in the presentation of findings, which resulted in not simply disagreement but anger and resentment. I conclude…

Institutional ethics review procedures aim -- in principle -- to minimise harm and evaluate risks, providing an important space to consider the safety of participants and researchers. However, literature has questioned the effectiveness of the process, particularly for reviewing 'risky' topics in a risk-averse environment. This article reports the…

In Comparative Judgement (CJ) exercises, examiners are asked to look at a selection of candidate scripts (with marks removed) and order them in terms of which they believe display the best quality. By including scripts from different examination sessions, the results of these exercises can be used to help with maintaining standards. Results from…

Background: Rapid reviews involve a streamlined approach to knowledge synthesis. They are used to identify high-quality evidence for the purpose of informing decisions and initiatives, completed over relatively short timeframes, and have been found to reach conclusions that do not differ extensively from full systematic reviews. Although common in…

Manually scoring and revising student essays has long been a time-consuming task for educators. With the rise of natural language processing techniques, automated essay scoring (AES) and automated essay revising (AER) have emerged to alleviate this burden. However, current AES and AER models require large amounts of training data and lack…