In this article, we argue that automated scoring engines should be transparent and construct relevant--that is, as much as is currently feasible. Many current automated scoring engines cannot achieve high degrees of scoring accuracy without allowing in some features that may not be easily explained and understood and may not be obviously and…

Quantitative measures of systematic differences in OSCE scoring across examiners (often termed examiner stringency) can threaten the validity of examination outcomes. Such effects are usually conceptualised and operationalised based solely on checklist/domain scores in a station, and global grades are not often used in this type of analysis. In…

Recent investigations on how people derive meaning from language have focused on task-dependent shifts between two cognitive systems. The symbolic (amodal) system represents meaning as the statistical relationships between words. The embodied (modal) system represents meaning through neurocognitive simulation of perceptual or sensorimotor systems…

After ChatGPT was released late in 2022, many arguments about its accuracy and use in education arose. In this article, I seek to provide evidence of the accuracy and validity of ChatGPT's responses to users' queries in education by applying a systematic review methodology to analyse publications in specific databases following PRISMA guidelines…

Meta-analysis of diagnostic test accuracy (DTA) is a powerful statistical method for synthesizing and evaluating the diagnostic capacity of medical tests and has been extensively used by clinical physicians and healthcare decision-makers. However, publication bias (PB) threatens the validity of meta-analysis of DTA. Some statistical methods have…

Categorical growth models describe examinee growth in terms of performance-level category transitions, which implies that some percentage of examinees will be misclassified. This paper introduces a new procedure for estimating the classification accuracy of categorical growth models, based on Rudner's classification accuracy index for item…

Heart rate samples (n = 4500-8000) from wearables were compared to electrocardiography during a steady-state ruck (Ruck-S), maximal effort ruck (Ruck-M), submaximal cycle (Cycle), and Tabata Circuit. One device was worn at each location (wrist: Polar Grit-X, Garmin Fenix 6; chest-straps: Polar H10, Garmin HRM-Pro; armband: Polar Verity).…

One of the challenges of discussing validity arguments for machine scoring of essays centers on the absence of a commonly held definition and theory of good writing. At best, the algorithms attempt to measure select attributes of writing and calibrate them against human ratings with the goal of accurate prediction of scores for new essays.…

Conversational skills, which are essential for effective social interactions and typically pose difficulties for individuals with autism spectrum disorder (ASD), include abilities such as initiating topics, engaging in back-and-forth dialog, and responding to conversational cues. Chatbots have been used in mental health fields, and the development…

This article provides a conceptual framework for considering both the theoretical and methodological factors that underpin the successful validation of a competency framework. Drawing on educational assessment literature, this article argues that a valid competency framework relates to an interpretive judgement of the credibility of the claims…

Purpose: Facial movement analysis during facial gestures and speech provides clinically useful information for assessing bulbar amyotrophic lateral sclerosis (ALS). However, current kinematic methods have limited clinical application due to the equipment costs. Recent advancements in consumer-grade hardware and machine/deep learning made it…

The questionnaire method has always been an important research method in psychology. The increasing prevalence of multidimensional trait measures in psychological research has led researchers to use longer questionnaires. However, questionnaires that are too long will inevitably reduce the quality of the completed questionnaires and the efficiency…

This article presents the consensus of an National Council on Measurement in Education Presidential Task Force on Foundational Competencies in Educational Measurement. Foundational competencies are those that support future development of additional professional and disciplinary competencies. The authors develop a framework for foundational…

To improve the validity of self-report measures, researchers should control for response style (RS) effects, which can be achieved with IRTree models. A traditional IRTree model considers a response as a combination of distinct decision-making processes, where the substantive trait affects the decision on response direction, while decisions about…

While noncompliance is a concerning challenging behavior and commonly reported by educators, its measurement is likely to be invalid and inaccurate given the subjectivity of the operational definition. Engagement is offered as a more valid, accurate measurement that may provide data regarding the amount of instruction accessed by the student. In…