This study investigates the validity and reliability of generative large language models (LLMs), specifically ChatGPT and Google's Bard, in grading student essays in higher education based on an analytical grading rubric. A total of 15 experienced English as a foreign language (EFL) instructors and two LLMs were asked to evaluate three student…

The performance decision tree (PDT; Fulcher et al., 2011) is a rubric style that is applicable to performance assessment, with origins in Upshur and Turner's (1995) empirically derived binary-choice, boundary-definition (EBB) scale. It is easier for raters to assess performance by evaluating multiple binary-choice descriptors. Additionally,…

Scoring rubrics are known to be effective for assessing writing for both testing and classroom teaching purposes. How raters interpret the descriptors in a rubric can significantly impact the subsequent final score, and further, the descriptors may also color a rater's judgment of a student's writing quality. Little is known, however, about how…

Both within the field of world language (WL) teacher education and across teacher education in other disciplines, critiques of the edTPA have increased over the past several years. In WL language education, scholars have identified issues related to raters' use of edTPA rubrics and a serious lack of transparency about rater expertise. To better…

Differential rater severity (DRS), one prevalent case of differential rater functioning (aka rater bias or rater interaction) effects, manifests itself when a rater assigns unusually severe or lenient ratings, threatening the validity and fairness of rater-mediated assessment. Building on a many-facets Rasch measurement (MFRM) approach, this study…

The study examined the use of the Oral Proficiency Interview (OPI) in assessing Chinese as a second language and the challenges faced by teachers in the Chinese Language Scholarship (CLS) program. The Concurrent Triangulation Mixed Method Research was used in the study in which qualitative and quantitative data are collected simultaneously,…

This large-scale study used a survey to collect data on K-12 world language program leadership and instructional practices in Massachusetts public schools, investigating associations between the presence of a program leader or primary evaluator who is a world language specialist and instructional practices, curriculum, and assessment. The study…

This research paper employs an exploratory framework to evaluate the potential of ChatGPT as an Automated Writing Evaluation (AWE) tool in teaching English as a Foreign Language (EFL) in Thailand. The main objective is to investigate how well ChatGPT can assess students' writing using prompts and pre-defined rubrics compared to human raters.…

Professional competence of pre-service English teachers needs to be done to find effective ways in improving their teaching performance, which requires the proper implementation of ICT during online learning. This study aims at investigating the level of professional competence of pre-service English teachers in vocational high school in applying…

Rating scales for writing assessment are critical in that they determine directly the quality and fairness of such performance tests. However, in many EFL contexts, rating scales are made, to certain extent, based on the intuition of teachers who strongly need a feasible and scientific route to guide their construction of rating scales. This study…

In writing assessment, finding a valid, reliable, and efficient scale is critical. Appropriate scales, increase rater reliability, and can also save time and money. This exploratory study compared the effects of a binary scale and an analytic scale across teacher raters and expert raters. The purpose of the study is to find out how different scale…

Assessing writing is a demanding task. If a lecturer of writing is not prepared with a reliable scoring rubric, the students' real performance might not be known. One of the well-known English as a second language (ESL) writing rubric is the Jacobs ESL Composition Profile which was developed by Jacobs, Zingraf, Wormuth, Hartfiel, & Hughey in…

Rater subjectivity has long been an intriguing topic. The use of discussion as a resolution method is a practical way to reduce this subjectivity. However, the efficacy of discussion depends on whether different raters get equally engaged in it or one rater tends to dominate others. This study investigated whether and how rater dominance occurs in…

Assessing language skills in an integrative form has drawn the attention of assessment experts in recent years. While some research data exists on integrative listening/reading-to-write assessment, there is comparatively little research literature on listening-to-speak integrated assessment. Also, little attention has been devoted to the role of…

In the present study, I examined the effects of rater characteristics, in particular, raters' familiarity with a foreign accent, on the assessment of second language (L2) pronunciation. Forty-three native English-speaking teachers were divided into three groups according to their reported types of familiarity with Korean accents: heritage,…