This paper presents an approach to standard setting that combines the prototype group method (PGM; Eckes, 2012) with a receiver operating characteristic (ROC) analysis. The combined PGM-ROC approach is applied to setting cut scores on a placement test of English as a foreign language (EFL). To implement the PGM, experts first named learners whom…

Two factors were investigated that are thought to contribute to consistency in rater scoring judgments: rater training and experience in scoring. Also considered were the relative effects of scoring rubrics and exemplars on rater performance. Experienced teachers of English (N = 20) scored recorded responses from the TOEFL iBT speaking test prior…

We addressed Deville and Chalhoub-Deville's (2006), Schoonen's (2012), and Xi and Mollaun's (2006) call for research into the contextual features that are considered related to person-by-task interactions in the framework of generalizability theory in two ways. First, we quantitatively synthesized the generalizability studies to determine the…

In language programs, it is crucial to place incoming students into appropriate levels to ensure that course curriculum and materials are well targeted to their learning needs. Deciding how and where to set cutscores on placement tests is thus of central importance to programs, but previous studies in educational measurement disagree as to which…

Elicited imitation (EI) has been widely used to examine second language (L2) proficiency and development and was an especially popular method in the 1970s and early 1980s. However, as the field embraced more communicative approaches to both instruction and assessment, the use of EI diminished, and the construct-related validity of EI scores as a…

This research inspects the allocation of involvement load to the evaluation component of the involvement load hypothesis, examining how three typical approaches to evaluation (cloze-exercises, sentence-writing, and composition-writing) promote word learning. The results of this research were partially consistent with the predictions of the…

Feedback, or information given to learners regarding their performance, is found to facilitate second language (L2) learning. Research also suggests that the timing of feedback (whether it is provided immediately or after a delay) may affect learning. The purpose of the present study was to identify the optimal feedback timing for L2 vocabulary…

The aim was to evaluate conceptual scoring of lexical organization in bilingual children with language impairment (BLI) and to compare BLI performance with monolingual children with language impairment (MLI). Word associations were assessed in 15 BLI and 9 MLI children. BLI were assessed in Arabic and Swedish, MLI in Swedish only. A number of…

This paper reports a post-hoc analysis of the influence of lexical difficulty of cue sentences on performance in an elicited imitation (EI) task to assess oral production skills for 645 child L2 English learners in instructional settings. This formed part of a large-scale investigation into effectiveness of foreign language teaching in Polish…

This paper describes how a Many Faceted Rasch Measurement (MFRM) approach can be applied to performance assessment focusing on rater analysis. The article provides an introduction to MFRM, a description of MFRM analysis procedures, and an example to illustrate how to examine the effects of various sources of variability on test takers' performance…

Automated scoring has the potential to dramatically reduce the time and costs associated with the assessment of complex skills such as writing, but its use must be validated against a variety of criteria for it to be accepted by test users and stakeholders. This study addresses two validity-related issues regarding the use of e-rater® with the…

This study investigates the effects of cognacy on vocabulary learning. The research expands on earlier designs by measuring learning of English-Japanese cognates with both decontextualized and contextualized tests, scoring responses at two levels of sensitivity, and examining learning in a more ecologically valid setting. The results indicated…

Adults demonstrate difficulty and pronounced variability when developing second language (L2) grammatical knowledge and reading skills. We examine explanations in terms of individual differences in working memory (WM). Despite numerous studies, the association between WM and adult second language (L2) acquisition remains unclear, and longitudinal…

This article reports on a study that investigated whether processing instruction(PI) or production-based instruction (PBI) is more effective for the teaching of regular past simple verb forms in English. In addition, this study examined whether explicit grammatical information (EI) mediates the effectiveness of PI or PBI. A total of 194 Turkish…

This research was conducted to find out whether or not using short stories significantly improve the speaking and writing achievements. A quasi-experimental study of non-equivalent pretest-posttest control group design or comparison group design was used in this research. The population of this research was the all first semester undergraduate…