This paper discusses the relevance of the discourse-based interview (DBI) for holistic research and access to more layers of meaning in the study of complex phenomena. We draw on our own work in workplace discourse and relevant research from different linguistic traditions, particularly sociolinguistic inquiry. We reflect on the potential of DBI…

It is widely believed that human rating performance is influenced by an array of different factors. Among these, rater-related variables such as experience, language background, perceptions, and attitudes have been mentioned. One of the important rater-related factors is the way the raters interact with the rating scales. In particular, how raters…

On the basis of the results of two prior studies at the US Naval Academy (USNA), which described the choice of study resources and the self-reported learning approaches of students of differing achievement levels, the current investigation examines how students of differing achievement levels in general chemistry actually solve multiple-choice…

This study used think-aloud protocols to compare essay-rating processes across holistic and analytic rating scales in the context of China's College English Test Band 6 (CET-6). A group of 9 experienced CET-6 raters scored the same batch of 10 CET-6 essays produced in an operational CET-6 administration twice, using both the CET-6 holistic…

The decision-making behaviors of 8 raters when scoring 39 persuasive and 39 narrative essays written by second language learners were examined, first using Rasch analysis and then, through think aloud protocols. Results based on Rasch analysis and think aloud protocols recorded by raters as they were scoring holistically and analytically suggested…

The amount of variability contributed to large-scale performance assessment scores by raters is a constant concern for those who wish to use results from these assessments for educational decisions. This study approaches the problem by examining the behaviors of essay scorers who demonstrate different levels of proficiency with a holistic scoring…