Purpose: The primary objective of this research paper was to examine the objectivity of the preselected items evaluation (PIE) method, a prevalent translation scoring method deployed by international institutions such as UAntwerpen, UGent and the University of Granada. Design/methodology/approach: This research critically analyzed the scientific…

The use of complexity-based evaluation methods remains relatively underexplored in the field of evaluation. While increasingly employed to assess complex interventions, Contribution Analysis (CA) continues to suffer from a lack of operationalization. In this article, we propose enhancing the implementation of CA by leveraging Participatory Systems…

As computer-based testing becomes more prevalent, the attention paid to response time (RT) in assessment practice and psychometric research correspondingly increases. This study explores the rate of Type I error in detecting preknowledge cheating behaviors, the power of the Kullback-Leibler (KL) divergence measure, and the L person fit statistic…

We are at a critical moment for assessment in schools. Teachers are called to navigate advances in classroom assessment research, top-down assessment policies, and lingering effects of the COVID-19 pandemic on teaching and learning. Embedded in this context are also systemic challenges to teachers' assessment practice. This paper analyses these…

Many researchers have evaluated how characteristics of feedback may influence trainee performance, but relatively little attention has been allocated to directly assessing trainee preference for feedback characteristics and its relation to performance. Thus, the primary purpose of this study was to use a within-subject experimental design to…

This reflective piece tells the story of how I started out doing Conversation Analysis (CA) and have been transitioning into doing mixed methods for some years now. My basic argument is that language learning talk is too complex a phenomenon to analyse using a single methodology. Specifically, it is extremely difficult to isolate from the…

This study evaluated the efficacy of two proposed methods in an operational standard-setting study conducted for a high-stakes language proficiency test of the U.S. government. The goal was to seek low-cost modifications to the existing Yes/No Angoff method to increase the validity and reliability of the recommended cut scores using a convergent…

Objective: The objective of this study was threefold: (a) to create a rubric for response complexity (RC), defined as an admixture of response length, grammatical diversity, categorisation, and sophistication; (b) to measure behavioural and cognitive engagement through students' written responses on a school-based written activity, and (c) to…

Background and Context: Feedback on one's progress is essential to new programming language learners, particularly in out-of-classroom settings. Though many study materials offer assessment mechanisms, most do not examine the accuracy of the feedback they deliver, nor give evidence on its validity. Objective: We investigate the potential use of a…

Rigor evaluation in mixed methods research is a growing need. Linking rigor to Hong and Pluye's (2019) concepts of methodological and reporting quality, the purpose of this article is to operationalize and expand Harrison et al.'s (2020) Rigorous Mixed Methods Framework. Drawing from a systematic methodological review of 66 inclusive education…

In response to a call for more research on evaluator education, this paper explores the use of role-play as an in-class activity for students in a graduate course in evaluation. We describe the activity in detail and use the Structure of Observed Learning Outcomes taxonomy as a tool to describe the cognitive complexity of student responses in the…

In selected-response assessments such as attitude surveys with Likert-type rating scales, examinees often select from rating scale categories to reflect their locations on a construct. Researchers have observed that some examinees exhibit "response styles," which are systematic patterns of responses in which examinees are more likely to…

This study examines the effect of item order (random, increasingly difficult, and decreasingly difficult) on student performance, test parameters, and student perceptions in multiple-choice tests administered in a paper-and-pencil format after online learning. In the research conducted using an explanatory sequential mixed methods design,…

Selecting texts that can support young readers is essential work for teachers because the 'right' text provides readers with opportunities to demonstrate their skills, strategies and comprehension. Text complexity guides offer teachers one way to select those texts, but despite these pedagogical supports, many readers continue to struggle with…

This article builds upon a proiminent definition of construct validity that focuses on variation in attributes causing variation in measurement outcomes. This article synthesizes the defintion and uses Rasch measurement modeling to explicate a modified conceptualization of construct validity for assessments of developmental attributes. If…