The present study explores two rating methods for peer assessment (analytical rating using criteria and comparative judgement) in light of concurrent validity, reliability and insufficient diagnosticity (i.e. the degree to which substandard work is recognised by the peer raters). During a second-year undergraduate course, students wrote a one-page…

Although there exists an extensive amount of research on subscores and their properties, limited research has been conducted on categorical subscores and their interpretations. In this paper, we focus on the claim of Feinberg and von Davier that categorical subscores are useful for remediation and instructional purposes. We investigate this claim…

Scoring high-dimensional assessments (e.g., > 15 traits) can be a challenging task. This paper introduces the multilabel neural network (MNN) as a scoring method for high-dimensional assessments. Additionally, it demonstrates how MNN can score the same test responses to maximize different performance metrics, such as accuracy, recall, or…

The proliferation of massive open online courses (MOOCs) highlights the necessity of developing accurate and diagnostic evaluation methods to assess the courses' quality and effectiveness. Hence, this study proposes a diagnostic MOOC evaluation (DME) method that combines the Analytic Hierarchy Process algorithm and learner review mining to…

Clinical Question: For young AAE speakers (P), how useful is the Developmental Sentence Scoring (DSS) compared with Index of Productive Syntax (IPSyn) in identifying developmental language disorder (DLD) in the presence of African American English (AAE)? Method: Structured Review. Study Sources: PsycInfo®, Education Source, Education Resources…

This study aims to develop a scale to measure the design thinking process and to evaluate the reliability and validity of this scale. It fills this gap by introducing a 36-item scale specifically designed to measure design thinking abilities across the five key stages of the design thinking process: empathize, define, ideate, prototype, and test,…

The purpose of this study was to investigate the Type I Error findings and power rates of the methods used to determine dimensionality in unidimensional and bidimensional psychological constructs for various conditions (characteristic of the distribution, sample size, length of the test, and interdimensional correlation) and to examine the joint…

Effective assessments guide student learning, refine teaching practices, ensure curriculum alignment, and foster workforce readiness. However, the emergence of generative artificial intelligence (GenAI) tools, such as ChatGPT, has significantly disrupted traditional assessment processes, raising concerns about academic integrity and necessitating…

Student Experience surveys are used to measure student attitudes towards their campus as well as to initiate conversations for institutional change. Validity evidence to support the interpretations of these surveys' results, however, is lacking. The first purpose of this study was to compare three Differential Item Functioning (DIF) methods on…

Purpose: The aim of this study was to advance the use of structured, monologic discourse analysis by validating an automated scoring procedure for core lexicon (CoreLex) using transcripts. Method: Forty-nine transcripts from persons with aphasia and 48 transcripts from persons with no brain injury were retrieved from the AphasiaBank database. Five…

The main problem of the educational evaluation validity is that it just copies the conceptual framework system of validity from educational measurement to its own conceptual system. The validity conceptual system that fits the need of theory and practice of educational evaluation has not been established yet. According to the inherent attributive…

All 50 states use observations to evaluate practicing teachers, but we know little about how administrators actually reason when they use those observation protocols. Drawing on think-aloud and stimulated recall data, this study describes the types of strategies and warrants practicing administrators used when rating with their district's…

Although much research has compared the functioning between analytic and holistic rating scales, little research has compared the functioning of binary rating scales with other types of rating scales. This quantitative study set out to preliminarily and comparatively validate binary and analytic rating scales intended for use in formative…

This work asks one simple question: "how reliable is the method used by the National Centre for Vocational Education Research (NCVER) to estimate projected rates of VET program completion?" In other words, how well do early projections align with actual completion rates some years later? Completion rates are simple to calculate with a…

The questionnaires' origin (sources from which elements of the questionnaire are derived) and initial development (process of making a questionnaire from elements) should be assessed before their measurement properties. There is no Critical Appraisal Tool (CAT) that comprehensively assesses the origin and initial development of questionnaires…