This article illustrates and differentiates the unique role cognitive interviews and think-aloud interviews play in developing and validating assessments. Specifically, we describe the use of (a) cognitive interviews to gather empirical evidence to support claims about the intended construct being measured and (b) think-aloud interviews to gather…

In this article, we explore the cognitive process and resources drawn upon when moderating artwork. The cognitive processes involved in the external moderation of non-exam assessments has received little attention; the few research studies that exist investigated moderation where the candidates' submissions were in mostly written form. No studies…

This article reports a validity study of History Assessments of Thinking (HATs), which are short, constructed-response assessments of historical thinking. In particular, this study focuses on aspects of cognitive validity, which is an examination of whether assessments tap the intended constructs. Think-aloud interviews with 26 high school…

Teacher observations are being used for high-stakes purposes in states across the country, and administrators often serve as raters in teacher evaluation systems. This paper examines how the cognitive aspects of administrators' use of an observation instrument, a modified version of Charlotte Danielson's Framework for Teaching, interact with the…

This case examines the affordances and limitations of using a case-study protocol to investigate teaching interns' (undergraduates studying to be teachers) knowledge of culturally relevant teaching practice. As researchers, we sought to understand whether, and the extent to which, the case-study protocol enables teacher-educators to measure…

Assessing pragmatic knowledge in the instructed setting is seen as a complex but necessary task, which requires the design of appropriate research methodologies to examine pragmatic performance. This study discusses the use of two different research methodologies, namely those of Discourse Completion Tests/Tasks (DCTs) and verbal reports. Research…

The purpose of this study was to identify and examine effective reading strategies used by adult deaf readers compared with student deaf readers. There were a total of 11 participants: 5 deaf adults ranging from 27 to 36 years and 6 deaf students ranging from 16 to 20 years. Assessment methods included interview and think-aloud procedures in which…

Findings derived from self-reported, structured survey questionnaires are commonly used in evaluation and applied research to inform policy-making and program decisions. Although there are a variety of issues related to the quality of survey evidence (e.g., sampling precision), the validity of response processes--how respondents process thoughts…

Self-regulated learning (SRL) with hypermedia environments involves a complex cycle of temporally unfolding cognitive and metacognitive processes that impact students' learning. We present several methodological issues related to treating SRL as an event and strengths and challenges of using online trace methodologies to detect, trace, model, and…

Age-related changes in the use of verbal processes for the efficient switching between tasks were investigated in 5-year-old children (N = 32, M age = 5.9 years) and 9-year-old children (N = 32, M age = 9.4 years). Task switching was assessed by means of a cued switching paradigm to examine two switching components: (a) to maintain and select and…

Introduction: Despite all the emphasis on metacognition, researchers currently use different techniques to assess metacognition. The purpose of this contribution is to help to clarify some of the paradigms on the evaluation of metacognition. In addition the paper reviews studies aiming to improve the learning process through metacognition. Method:…

Studies what inferences test-takers make as they struggle to match their expectations for an answer with the choices offered in a multiple-choice question. Finds that the 27 types of inferences observed within the metacognitive, cognitive, and affective interactions displayed during these protocols suggested a wealth of activity and thought. (MG)

The purpose of this article is to provide teachers with examples of how to perform effective think-alouds. Twelve think-alouds are described. A rationale for their use in kindergarten through grade 5 is also included. In addition, this article was designed to provide teachers with lessons and activities to assist students in developing the ability…

This report characterizes human problem solving in digital circuit design. Protocols of 11 different designers with varying degrees of training were analyzed by identifying the designers' problem solving strategies and discussing activity patterns that differentiate the designers. These methods are proposed as a tentative basis for assessing…

Thirty-one students (ages 8-14) with learning disabilities or behavior disorders solved arithmetic word problems while a narrative protocol was developed. Eight types of cognitive processing difficulties were identified (such as lack of self-correction and inclusion of extraneous information), and the utility of protocol analysis as an assessment…