This study investigated factors that could affect inter-examiner reliability in the pronunciation assessment component of speaking tests. We hypothesized that the rating of pronunciation is susceptible to variation in assessment due to the amount of exposure examiners have to nonnative English accents. An inter-rater variability analysis was…

The increasing influence of sociocultural theories of learning on assessment practices in second language education necessitates an expansion of the knowledge base that teacher-assessors need to develop (what teachers need to know) and related changes in the processes of language teacher education (how they learn and develop it). Teacher assessors…

The Modern Language Aptitude Test (Paper-and-Pencil Version, henceforth, the MLAT) measures "an individual's ability to learn a foreign language." It targets English-speaking adults (over Grade 9) who are literate. The test has only one form, which has not changed since it was first published by the Psychological Corporation in 1959. The test can…

Previous test fairness frameworks have greatly expanded the scope of fairness, but do not provide a means to fully integrate fairness investigations and set priorities. This article proposes an approach to guide practitioners on fairness research and practices. This approach treats fairness as an aspect of validity and conceptualizes it as…

Despite the growing interest of the language testing community in standard setting, primarily due to the use of the Common European Framework of Reference (CEFR-Council of Europe, 2001), the participants' decision-making process in the CEFR standard setting context remains unexplored. This study attempts to fill in this gap by analyzing these…

Testing of second language pragmatic competence is an underexplored but growing area of second language assessment. Tests have focused on assessing learners' sociopragmatic and pragmalinguistic abilities but the speech act framework informing most current productive testing instruments in interlanguage pragmatics has been criticized for…

The purpose of this study was to explore whether the results of Tree Based Regression (TBR) analyses, informed by a validated cognitive model, would enhance the interpretation of item difficulties in terms of the cognitive processes involved in answering the reading items included in two forms of the Michigan English Language Assessment Battery…

"Pseudowords", or non-real words, were introduced to the Yes/No (YN) vocabulary test format to provide a means of checking for overestimation of word knowledge by test takers. The purpose of this study is to assess the assumption that more pseudoword checks (false alarms) indicate more instances of overestimation of word knowledge in YN…

It is the consensus that, as a result of the extra constraints placed on working memory, texts written in a second language (L2) are usually of lower quality than texts written in the first language (L1) by the same writer. However, no method is currently available for quantifying the quality difference between L1 and L2 texts. In the present…

Raters are central to writing performance assessment, and rater development--training, experience, and expertise--involves a temporal dimension. However, few studies have examined new and experienced raters' rating performance longitudinally over multiple time points. This study uses operational data from the writing section of the MELAB (n =…

Rating scale design and development for testing speaking is generally conducted using one of two approaches: the measurement-driven approach or the performance data-driven approach. The measurement-driven approach prioritizes the ordering of descriptors onto a single scale. Meaning is derived from the scaling methodology and the agreement of…

The effectiveness of individualized feedback on rater behavior has been investigated in several previous studies (e.g. Elder, Knoch, Barkhuizen, & von Randow, 2005; Lunt, Morton, & Wigglesworth, 1994; O'Sullivan & Rignall, 2007; Wigglesworth, 1993). The findings of these studies are somewhat inconclusive. However, all of these studies investigated…

This paper reports on an investigation of the potential for a shared-L1 advantage on an academic English listening test featuring speakers with L2 accents. Two hundred and twelve second-language listeners (including 70 Mandarin Chinese L1 listeners and 60 Japanese L1 listeners) completed three versions of the University Test of English as a Second…

This study examined the relationship between scores on the TOEFL Internet-Based Test (TOEFL iBT[R]) and academic performance in higher education, defined here in terms of grade point average (GPA). The academic records for 2594 undergraduate and graduate students were collected from 10 universities in the United States. The data consisted of…

Scores assigned by trained raters and by an automated scoring system (SpeechRater[TM]) on the speaking section of the TOEFL iBT[TM] were validated against a communicative competence criterion. Specifically, a sample of 555 undergraduate students listened to speech samples from 184 examinees who took the Test of English as a Foreign Language…