This study conducts a comprehensive quantitative evaluation of OpenAI's language model, ChatGPT 4, for grading Task 2 writing of the IELTS exam. The objective is to assess the alignment between ChatGPT's grading and that of official human raters. The analysis encompassed a multifaceted approach, including a comparison of means and reliability…

Studies on automatic scoring systems in writing assessments have also evaluated the relationship between human and machine scores for the reliability of automated essay scoring systems. This study investigated the magnitudes of indices for inter-rater agreement and discrepancy, especially regarding human and machine scoring, in writing assessment.…

This study explored two assessment approaches to oral performances: analytical complexity, accuracy, and fluency (CAF) indices and human raters' evaluations. CAF indices are frequently used in second-language speaking (L2) research; however, because tasks are communicative and goal-oriented, the degree to which students achieve such communicative…

Whereas many scholars have emphasized the relative importance of "comprehensibility" as an ecologically valid goal for L2 speech training, testing, and development, eliciting listeners' judgments is time-consuming. Following calls for research on more efficient L2 speech rating methods in applied linguistics, and growing attention toward…

The quality of writing in a second language (L2) is one of the indicators of the level of proficiency for many college students to be eligible for departmental studies. Although certain software programs, such as Intelligent Essay Assessor or IntelliMetric, have been introduced to evaluate second-language writing quality, an overall assessment of…

Previous research that explores the effect of ethnicity in the perception of speaker accentedness and personality traits often finds that Asian appearance contributes to a more accented and less competent impression. Importantly, most of the work done to date employed only Caucasian first language-speaking listeners; moreover, ethnicity and gender…

This article investigates the central role of topic in the IELTS Speaking Test (IST). Topic has developed a dual personality in this interactional setting: topic-as-script is the scripted statement of topic on the examiner's cards prior to the interaction, whereas topic-as-action is how topic is developed by the candidate during the course of the…

This large-scale study used a survey to collect data on K-12 world language program leadership and instructional practices in Massachusetts public schools, investigating associations between the presence of a program leader or primary evaluator who is a world language specialist and instructional practices, curriculum, and assessment. The study…

Drawing on Glenn Fulcher's extensive work in performance-based language assessment of speaking, this paper explores the assessment of L2 speaking ability in local language testing contexts. For that purpose, I review Fulcher's influential work that highlights the relationship between the speaking construct, the task, the performance, and the…

This study examined how L2 English speakers interpreted the notion of native English speakers (NESs) and nonnative English speakers (NNESs) and whether nativeness would influence their self-perception and speech production. It aimed at filling the following research gaps. First, limited studies have explored how L2 English speakers view the other…

This study explicates the nature of second language (L2) pragmatic interaction focusing on the quantitative function of interactional features. A relationship between the fine-grained interactional features elicited from learners' role-play performances at varying levels and trained raters' scores was investigated. The corpus of 102 learners'…

This study compared holistic and analytic marking methods for their effects on parameter estimation (of examinees, raters, and items) and rater cognition in assessing speech act production in L2 Chinese. Seventy American learners of Chinese completed an oral Discourse Completion Test assessing requests and refusals. Four first-language (L1)…

This study examined the use of a popular Automatic Speech Recognition (ASR), Google Voice Typing (GVT), to automatically assess English as second language pronunciation. It aimed to answer the following question: What is the relationship between GVT-rated scores and human-rated scores? To answer this question, we compared audio recordings of 56…

This mixed methods study examined holistic, analytic, and part marking models (MMs) in terms of their measurement properties and impact on candidate CEFR classifications in a semi-direct online speaking test. Speaking performances of 240 candidates were first marked holistically and by part (phase 1). On the basis of phase 1 findings--which…

The factors that influence rater scoring have been a subject of great interest to researchers in second language assessment. However, the research on the impact of test-takers' speech profiles (e.g., a jagged or a flat profile reflecting analytic subscores) on raters' scoring behaviors remains to be seen. To investigate the role of speech profiles…