A critical component of cognitive diagnostic models (CDMs) is a Q-matrix that stipulates associations between items of a test and their required attributes. The present study aims to develop and empirically validate a Q-matrix for the listening comprehension section of the International English Language Testing System (IELTS). To this end, a…

Although Diagnostic Classification Models (DCMs) were introduced to education system decades ago, it seems that these models were not employed for the original aims upon which they had been designed. Using DCMs has been mostly common in analyzing large-scale non-diagnostic tests and these models have been rarely used in developing Cognitive…

This article explores the issue of identifying guessers -- with a specific focus on multiple-choice tests. Guessing has long been considered a problem due to the fact that it compromises validity. A test taker scoring higher than they should through guessing does not provide a picture of their actual ability. After an initial description of issues…

The purpose of this study is to equate and further validate three forms of the vocabulary size test (VST) created by Aizawa and Mochizuki (2010). These three forms, VST 1, 2, and 3, were administered to a cohort of 189 high school students ranging in age from 16 to 18 in April of their 1st, 2nd, and 3rd year of high school. Although these…

Purpose: Our aim was to evaluate the psychometric properties of the online administered format of the Test of Narrative Language--Second Edition (TNL-2; Gillam & Pearson, 2017), given the importance of assessing children's narrative ability and considerable absence of psychometric studies of spoken language assessments administered online.…

Cognitive diagnostic models (CDMs) have received sustained attention in educational settings because they can be used to operationalize formative assessment to provide diagnostic feedback and inform instruction. A large number of CDMs have been developed over the past few years. An important component of all CDMs is a Q-matrix that specifies a…

Purpose: Our aim was to evaluate the psychometric properties of the online administered format of the Test of Narrative Language--Second Edition (TNL-2; Gillam & Pearson, 2017), given the importance of assessing children's narrative ability and considerable absence of psychometric studies of spoken language assessments administered online.…

For test scores that rely on the accurate estimation of ability via an IRT model, their use and interpretation is dependent upon the assumption that the IRT model fits the data. Examinees who do not put forth full effort in answering test questions, have prior knowledge of test content, or do not approach a test with the intent of answering…

This paper examines how different test methods may tap different aspects of second language knowledge. It employs multiple-choice (MC) and constructed response (CR) items which yield distinct or convergent information in the computer delivered testing of English in its presentation of this factor. In order to examine the effects of test method, a…

Validated under a Rasch framework (Beglar, 2010), the Vocabulary Size Test (VST) (Nation & Beglar, 2007) is an increasingly popular measure of decontextualized written receptive vocabulary size in the field of second language acquisition. However, although the validation indicates that the test has high internal reliability, still unaddressed…

This study explores the attitudes of raters of English speaking tests towards the global spread of English and the challenges in rating speakers of Indian English in descriptive speaking tasks. The claims put forward by language attitude studies indicate a validity issue in English speaking tests: listeners tend to hold negative attitudes towards…

The paper by Wilson and Moore (this volume), based on the Messick Lecture delivered in 2006 at the annual Language Testing Research Colloquium in Melbourne, may present a familiar challenge to some language testers: of reading outside one's comfort zone. The distinctive character of language testing lies in its combination of two primary fields of…

This study examined the construct validity of the TOEFL iBT Speaking subsection for the purposes of international teaching assistant (ITA) certification, a purpose for which it was not specifically designed. The factor structure of the new TOEFL was compared with that of another language performance test in use at a major American research…

Despite the importance of having a reliable and valid measure of Second Language (L2) proficiency, L2 researchers of less commonly taught languages rarely have such a tool. Existing proficiency measures (e.g., DLPT, OPI) are often costly, labor-intensive, time-consuming, or unavailable to the public. With the intent to provide a practical and…

The primary purpose of this study was to provide preliminary validity evidence for a 140-item form of the Vocabulary Size Test, which is designed to measure written receptive knowledge of the first 14,000 words of English. Nineteen native speakers of English and 178 native speakers of Japanese participated in the study. Analyses based on the Rasch…