This paper presents an approach to standard setting that combines the prototype group method (PGM; Eckes, 2012) with a receiver operating characteristic (ROC) analysis. The combined PGM-ROC approach is applied to setting cut scores on a placement test of English as a foreign language (EFL). To implement the PGM, experts first named learners whom…

Two factors were investigated that are thought to contribute to consistency in rater scoring judgments: rater training and experience in scoring. Also considered were the relative effects of scoring rubrics and exemplars on rater performance. Experienced teachers of English (N = 20) scored recorded responses from the TOEFL iBT speaking test prior…

We addressed Deville and Chalhoub-Deville's (2006), Schoonen's (2012), and Xi and Mollaun's (2006) call for research into the contextual features that are considered related to person-by-task interactions in the framework of generalizability theory in two ways. First, we quantitatively synthesized the generalizability studies to determine the…

Elicited imitation (EI) has been widely used to examine second language (L2) proficiency and development and was an especially popular method in the 1970s and early 1980s. However, as the field embraced more communicative approaches to both instruction and assessment, the use of EI diminished, and the construct-related validity of EI scores as a…

In language programs, it is crucial to place incoming students into appropriate levels to ensure that course curriculum and materials are well targeted to their learning needs. Deciding how and where to set cutscores on placement tests is thus of central importance to programs, but previous studies in educational measurement disagree as to which…

This paper reports a post-hoc analysis of the influence of lexical difficulty of cue sentences on performance in an elicited imitation (EI) task to assess oral production skills for 645 child L2 English learners in instructional settings. This formed part of a large-scale investigation into effectiveness of foreign language teaching in Polish…

Perceptual (mis)matches between teachers and learners are said to affect learning success or failure. Self-assessment, as a formative assessment tool, may, inter alia, be considered a means to minimize such mismatches. Therefore, the present study investigated the extent to which learners' assessment of their own speaking performance, before and…

This paper reports an investigation into how the prompt may influence the discourse of group oral tests. The group oral test, in which three or four participants are rated on their ability to discuss a prompt, is a format for assessing the spoken ability of language learners. In this study, 141 Japanese university students were videoed in 41 group…

Language assessment courses (LACs) are taught by professionals who have majored in the area of language testing (language testers or LTs), but also by others who come from different language-related majors (non-language testers, non-LTs). Different language assessment courses may be developed, depending on who teaches the course and the…

Motivated by cognitive theories of graph comprehension, this study systematically manipulated characteristics of a line graph description task in a speaking test in ways to mitigate the influence of graph familiarity, a potential source of construct-irrelevant variance. It extends Xi (2005), which found that the differences in holistic scores on…

Although the Word Associates Format (WAF) is becoming more frequently used as a depth-of-knowledge measure, relatively little validation has been carried out on it. This report of two validation studies tackles various important WAF issues yet to be satisfactorily resolved. Study 1 conducted introspective interviews regarding students' WAF…

This study investigated the validity of four theoretically motivated traits of writing ability across English and Korean, based on elementary school students' responses to letter- and story-writing tasks. Their responses were scored analytically and analyzed using confirmatory factor analysis. The findings include the following. A model of writing…

Practitioners working closely with second language (L2) writers in the US recognize at least two types of L2 students: international (IL2) and Generation 1.5 (G1.5) students. Some argue that specific differences in each group's writing performance are evident (cf. Harklau, 2003; Reid, 2006); however, investigations into observable and measurable…