Automated scoring of student responses to open-ended questions, including short-answer questions, has great potential to scale to a large number of responses. Recent approaches for automated scoring rely on supervised learning, i.e., training classifiers or fine-tuning language models on a small number of responses with human-provided score…

Examiners' judgements play a critical role in competency-based assessments such as objective structured clinical examinations (OSCEs). The standardised nature of OSCEs and their alignment with regulatory accountability assure their wide use as high-stakes assessment in medical education. Research into examiner behaviours has predominantly explored…

As part of its "Guiding Principles for Evaluators," the American Evaluation Association (AEA) requires that evaluators develop cultural competencies. Using a successive-independent-samples design, the researchers sought to compare perceptions of cultural competence across a duration of 10 years. Qualitative data were collected via online…

It is widely believed that raters' cognition is an important aspect of writing assessment, as it has both logical and temporal priority over scores. Based on a critical review of previous research in this area, it is found that raters' cognition can be boiled to two fundamental issues: building text images and strategies for articulating scores.…

The primary purpose of Institutional Review Boards (IRBs) is to protect the rights and welfare of human research participants. Evaluation and measurement of how IRBs satisfy this purpose and other important goals are open questions that demand empirical research. Research on IRBs, and the Human Research Protection Programs (HRPPs) of which they…

This replication study seeks to extend the generalizability of an exploratory study (McDonough et al., 2019) that identified holds (i.e., temporary cessation of dynamic movement by the listener) as a reliable visual cue of non-understanding. Conversations between second language (L2) English speakers in the Corpus of English as a Lingua Franca…

The recent Next Generation Science Standards recommend students have increased opportunities engaging in modeling work scientists do. Science classroom tasks include simple inquiry compared to more complex authentic inquiry that closely resembles authentic science. Science fair participation presents students with the opportunity to engage in…

Rater fit analyses provide insight into the degree to which rater judgments correspond to expected properties, as defined within a measurement framework. Parametric models such as the Rasch model provide a useful framework for evaluating rating quality; however, these models are not appropriate for all assessment contexts. The purpose of this…

This essay provides advice for effectively reviewing conference proposals, including how to write comments that are helpful to proposal authors, how to use the "Comments to Program Chair" box, and issues to consider when assigning proposal ratings and recommending acceptance or rejection. Several benefits of reviewing proposals are…

Human Resource (HR) practices like performance appraisal (PA) training are meant to ensure that employees are equipped with the knowledge and skills needed for the attainment of organisational goals. However, gaps still exist on the relationship between PA employees' training and employees' performance. This study aimed to establish the…

This study aimed to examine the potentials of the NLP approach in detecting discourse markers (DMs), namely okay, in transcribed spoken data. One hundred thirty-eight concordance lines were presented to human referees to judge the functions of okay in them as a DM or Non-DM. After that, the researchers used a Python script written according to the…

Listener-based ratings have become a prominent means of defining second language (L2) users' global speaking ability. In most cases, local listeners are recruited to evaluate speech samples in person. However, in many teaching and research contexts, recruiting local listeners may not be possible or advisable. The goal of this study was to hone a…

This guidance is intended for the planning stage of trials -- in particular, trials for which the primary outcome measure being considered is attainment at GCSE (and in particular, GCSE English Language and Mathematics). However, it is generalisable to any other graded qualification or assessment used as an outcome measure, i.e. A-levels. It is…

In a randomized trial that collects text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by human raters. An impact analysis can then be conducted to compare treatment and control groups, using the hand-coded scores as a measured outcome. This…

Experiential education partnered with guided reflection is thought to support students with higher-order thinking skills. In this study, 44 reflections from two university-level sustainability courses were compared. In both courses students were asked to write a reflection, but only one course used the Reflective Learning Framework (RLF). Tests of…