This study evaluated the efficacy of two proposed methods in an operational standard-setting study conducted for a high-stakes language proficiency test of the U.S. government. The goal was to seek low-cost modifications to the existing Yes/No Angoff method to increase the validity and reliability of the recommended cut scores using a convergent…

Over the past decade, testing and assessing spoken-language interpreting has garnered an increasing amount of attention from stakeholders in interpreter education, professional certification, and interpreting research. This is because in these fields assessment results provide a critical evidential basis for high-stakes decisions, such as the…

As access to smartphones and emerging technologies has become ubiquitous in our daily lives and in language learning, technology-mediated social interaction has become common in teaching and assessing L2 speaking. The changing ecology of L2 spoken interaction provides language educators and testers with opportunities for renewed test design and…

Despite the increased use of integrated tasks in high-stakes academic writing assessment, research on rating criteria which reflect the unique construct of integrated summary writing skills is comparatively rare. Using a mixed-method approach of expert judgement, text analysis, and statistical analysis, this study examines writing features that…

The quality of sign language interpreting (SLI) is a gripping construct among practitioners, educators and researchers, calling for reliable and valid assessment. There has been a diverse array of methods in the extant literature to measure SLI quality, ranging from traditional error analysis to recent rubric scoring. In this study, we want to…

In recent years, continuing advances in technology have increased the capacity to automate the extraction of a range of linguistic features of texts and thus have provided the impetus for the substantial growth of corpus linguistics. While corpus linguistic tools and methods have been used extensively in second language learning research, they…

Critical language assessment (CLA) has been addressed in numerous studies. However, the majority of the studies have overlooked the need for a practical framework to measure the CLA dimension of teachers' language assessment literacy (LAL). This gap prompted us to develop and validate a critical language assessment literacy (CLAL) scale to further…

This study investigates the validity of assessing L2 pragmatics in interaction using mixed methods, focusing on the evaluation inference. Open role-plays that are meaningful and relevant to the stakeholders in an English for Academic Purposes context were developed for classroom assessment. For meaningful score interpretations and accurate…

With recent statistical advances in cognitive diagnostic assessment (CDA), the CDA approach has been increasingly applied to non-diagnostic tests partly to meet accountability demands for student achievement. The study aimed to evaluate critically the validity of the CDA application to an existing non-diagnostic L2 reading comprehension test and…

Speech act theory-based, second language pragmatics testing (SLPT) poses problems for validation due to a lack of correspondence with empirical conversational data. Since conversation analysis (CA) provides a richer and more accurate account of language behavior, it may be preferred as a basis for SLPT development. However, applying CA methodology…

Argues that a growing awareness of the fundamentally social character of language assessment challenges a rethinking of priorities and responsibilities in language testing research. This awareness is the result of the treatment of the social character of educational assessment in Messick's (1989) work on validity and by actual changes triggered by…

Considers the ecological validity of tests in predicting the effects of speech and language impairment on communicative performance in real-world settings and the need for systematic observation of communicative behaviors, particularly of individuals from other cultures. Discusses whether standardized testing and systematic observation are…

This article reports on a collaborative study involving ESL teachers in an Australian English Language Centre as they work through some of their concerns about reliability and validity in their assessment practices. The focus of this article is on how teachers work with the Curriculum Standards Framework (CSF) as an assessment tool. The discussion…