This chapter reports on a study investigating non-expert raters' scoring behavior and cognitive processes involved in evaluating speech acts and pragmatic routines in L2 Chinese. Pragmatic production data were collected from 51 American learners of Chinese, who completed a 12-item oral Discourse Completion Test (DCT). The learners were divided…

This study examines the feasibility and potential advantages of using large language models, in particular GPT-4o, to perform partial credit grading of large numbers of student written responses to introductory level physics problems. Students were instructed to write down verbal explanations of their reasoning process when solving one conceptual…

Social justice evaluation approaches advance antioppressive and culturally responsive methods to mitigate societal inequalities. However, social justice efforts will fall short if they focus solely on one side of inequality: oppression. Accordingly, this article argues that the other side of inequality--privilege--needs to be scrutinized. This…

Since its inception in 1993, Ofsted has been charged with maintaining standards in schools in England. But how has this worked out? A former inspector gives his account of the journey from the inside. His point of view can be summarised from a conversation he had with a senior Ofsted inspector on his first Inspection: 'You're still on their side.…

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two…

Systematic reviews are vital for guiding practice, research and policy, although they are often slow and labour-intensive. Large language models (LLMs) could speed up and automate systematic reviews, but their performance in such tasks has yet to be comprehensively evaluated against humans, and no study has tested Generative Pre-Trained…

This study examined the perspectives of accreditation liaison officers, institutional accreditor peer-reviewers, and chief library officer perspectives on the role that the academic librarian, academic library, and information literacy should play in institutional accreditation. The study was framed using a conceptual model based on New…

Ethics review processes are frequently perceived as extending from codes and protocols rooted in biomedical disciplines. As a result, many researchers in the humanities and social sciences (HASS) find these processes to be misaligned, if not outrightly obstructive to their research. This leads some scholars to advocate against HASS participation…

Engaging students in creating learning resources has demonstrated pedagogical benefits. However, to effectively utilize a repository of student-generated content (SGC), a selection process is needed to separate high- from low-quality resources as some of the resources created by students can be ineffective, inappropriate, or incorrect. A common…

This paper explores the regulators' perspective and demonstrates how legitimacy deficits of private universities outweigh performance results in decisions regarding university inspections. We examined the period when the regulator had an urgent claim on Russian universities, particularly during the campaign to 'clean the system of higher…

A budding area of research is devoted to studying evaluator curriculum, yet to date, it has focused exclusively on describing the content and emphasis of topics or competencies in university-based programs. This study aims to expand the foci of research efforts and investigates the extent to which evaluators agree on what competencies should guide…

Performance standards have important consequences for all the stakeholders in the assessment of L2 academic writing. These standards not only describe the level of writing performance but also provide a basis for making evaluative decisions on the academic writing. Such a high-stakes role of the performance standards requires the enhancement of…

Assessment practitioners in higher education follow a variety of paths to their roles. Diverse preparation supports creative problem-solving in a changing educational landscape. However, it can also lead to inconsistency in language, preparation, and background knowledge. Further, the chasms between assessment practitioners' paths can lead to…

The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance…

In making validity arguments, a central consideration is whether the instrument fairly and adequately covers intended content, and this is often evaluated by experts. While common procedures exist for quantitatively assessing this, the effect of loss aversion--a cognitive bias that would predict a tendency to retain items--on these procedures has…