This study investigates the reliability and consistency of a custom GPT-based scoring system in comparison to trained human raters, focusing on B1-level opinion paragraphs written by English preparatory students. Addressing the limited evidence on how AI scoring systems align with human evaluations in foreign language contexts, the study provides…

This study developed and validated a Decision-Making Skill Test (DMST) for Turkish adolescents to address the lack of culturally appropriate assessment tools for multi-criteria decision-making skills. A cross-sectional design was employed with 427 participants aged 11-17 years from diverse socioeconomic backgrounds across Turkey. Following…

Open access to novel AI tools offers unprecedented opportunities for human-AI collaboration in writing instruction and assessment. While research on using generative AI tools like ChatGPT in these contexts is emerging, more is needed to understand their effectiveness as Automated Writing Evaluation (AWE) tools. This study explores the potential of…

The Educative Teacher Performance Assessment (edTPA) is a system of standardized portfolio assessments of teaching performance mandated for use by educator preparation programs in 18 states, and approved in 21 others, as part of initial certification for preservice teachers. Because of the high stakes involved for examinees, it is critical that…

This meta-analysis explores the correlation between self-assessment (SA) and language performance. Sixty-seven studies with 97 independent samples involving more than 68,500 participants were included in our analysis. It was found that the overall correlation between SA and language performance was 0.466 (p < 0.01). Moderator analysis was…

Reliability and validity are two important topics that have been studied for many decades in the educational measurement field, including discussions of Writing Studies' subfield of writing assessment, since the establishment of the College Entrance Exam Board [CEEB] in 1899 (Huot et al., 2010). In recent years, scholarly conversations of fairness…

We analyzed data from the WASI-II manual to determine discrepancy score reliabilities of the Verbal Comprehension (VCI) and Perceptual Reasoning (PRI) indexes and the four subtests in the child and adult standardization samples. Reliabilities of the VCI-PRI discrepancy scores range from 0.78 to 0.86 for children and 0.82 to 0.89 for adults and…

This article presents some equivalent forms of the common Kuder-Richardson Formula 21 and 20 estimators for nondichotomous data belonging to certain other exponential families, such as Poisson count data, exponential data, or geometric counts of trials until failure. Using the generalized framework of Foster (2020), an equation for the reliability…

The author should give careful consideration to the study eligibility criteria of systematic reviews (SRs) and follow it after review protocol development to reduce the possibility of manipulation of inclusion. Our aim was to investigate the prevalence of differences in study eligibility criteria between non-Cochrane SRs and their pre-registered…

The population discrepancy between unstandardized and standardized reliability of homogeneous multicomponent measuring instruments is examined. Within a latent variable modeling framework, it is shown that the standardized reliability coefficient for unidimensional scales can be markedly higher than the corresponding unstandardized reliability…

Coefficient omega indices are model-based composite reliability estimates that have become increasingly popular. A coefficient omega index estimates how reliably an observed composite score measures a target construct as represented by a factor in a factor-analysis model; as such, the accuracy of omega estimates is likely to depend on correct…

Purpose: This study compared the interrater reliability of adult naïve listeners' perceptual assessments of different speech variables in children with a cleft palate with or without a cleft lip (CP ± L). In addition, the study investigated whether the listeners were able to perceive differences in these speech variables before and after speech…

Internalizing and externalizing problems are common targets for school mental health screening. Prior research supports the interpretation of scores from the Youth Internalizing Problems Screener (YIPS) and the Youth Externalizing Problems Screener (YEPS), which were developed separately yet intended as companion measures. We extended previous…

Ensuring consistent and reliable scoring is paramount in education, especially in performance-based assessments. This study delves into the critical issue of marking consistency, focusing on speaking proficiency tests in English language learning, which often face greater reliability challenges. While existing literature has explored various…

As there is a paucity of instrument investigating a hybrid teaching conception, the current study is seen as part of attempt to fill this gap. The subjects in the study were 310 University participants--instructors in Socialist Republic of Viet Nam (Vietnam). The survey was implemented with the use of Cognitive Constructivism-oriented Teaching…