The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance…

Multistage adaptive testing (MST) has become popular in the testing industry because the research has shown that it combines the advantages of both linear tests and item-level computer adaptive testing (CAT). The previous research efforts primarily focused on MST design issues such as panel design, module length, test length, distribution of test…

Background: Non-word repetition (NWR) tests are an important way speech and language therapists (SaLTs) assess language development. NWR tests are often scored whilst participants make their responses (i.e., in real time) in clinical and research reports (documented here via a secondary analysis of a published systematic review). Aims: The main…

To improve clinical teaching skills, feedback on teachers' strengths and weaknesses needs to be reliable, timely, and relevant. To provide timely feedback we undertook development of an analytical dashboard to provide learner feedback to our faculty. As dashboard data displays are limited we performed a modified Delphi (mDelphi) method to…

Computational thinking (CT) is a multidimensional term that encompasses a wide variety of problem-solving skills related to the field of computer science. Unfortunately, standardized, valid, and reliable methods to assess CT skills in preschool children are lacking, compromising the reliability of the results reported in CT interventions. To…

This study investigates the application of the fuzzy logic method for scoring open-ended items, specifically comparing its effectiveness against traditional scoring methods. Utilizing the fuzzy TOPSIS method within the mathematics domain, this research established seven criteria for evaluating open-ended responses, developed in consultation with…

How do we make judgements about the quality of online courses? Checklists and rubrics are commonplace in higher education for establishing and measuring design features of online courses. They are created and used by institutions, academics and educational designers to standardise measures for quality online course design. Despite an intensifying…

Despite their widespread adoption and recognised benefits, rubrics have been critiqued for their potential misalignment with student needs. The voices of international students, who constitute a substantial portion of the higher education population, remain underrepresented. This study examines the perspectives of international undergraduate…

Validation is needed for any newly developed model or framework because it requires several real-life applications. The investment made into e-learning in medical education is daunting, as is the expectation for a positive return on investment. The medical education domain requires data-wise implementation of e-learning as the debate continues…

Machine learning methods can be used to reduce the manual workload in exam grading, making it possible for teachers to spend more time on other tasks. However, when it comes to grading exams, fully eliminating manual work is not yet possible even with very accurate automated grading, as any grading mistakes could have significant consequences for…

This study explores how assessment criteria are applied in grading student work. It is found that explicit assessment criteria do not work as authoritative guidance as expected and that tacit criteria are more decisive in awarding a certain grade. Various sources that form idiosyncratic tacit criteria are identified. These sources, including…

The Person-in-the-Rain (PITR) drawing scoring system primarily assesses stress, excluding consideration of color usage. In contrast, the Formal Elements Art Therapy Scale (FEATS) effectively evaluates psychopathological disorders and provides a comprehensive assessment of color usage. This study aimed to: (1) develop an alternative scoring system…

In this study, it was aimed to develop a rubric to evaluate the creative story writing skill levels of seventh grade secondary school students. The research was designed in quantitative research method and survey model. In the research, convenience sampling technique was used and 270 students studying at the seventh grade level of secondary school…

Automatic essay scoring (AES) is an essential educational application in natural language processing. This automated process will alleviate the burden by increasing the reliability and consistency of the assessment. With the advances in text embedding libraries and neural network models, AES systems achieved good results in terms of accuracy.…

Background: In a randomized trial that collects text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by human raters. An impact analysis can then be conducted to compare treatment and control groups, using the hand-coded scores as a measured outcome.…