This case study describes how our research team conducted the qualitative think-aloud approach (or cognitive pretesting) to obtaining cognitive validity evidence for a biology inference-making and reasoning measure for undergraduate students. The main goal of our work was to gather high-quality student think-aloud data of reasoning while they were…

This article reports a validity study of History Assessments of Thinking (HATs), which are short, constructed-response assessments of historical thinking. In particular, this study focuses on aspects of cognitive validity, which is an examination of whether assessments tap the intended constructs. Think-aloud interviews with 26 high school…

Across education, certification and licensure, there are repeated calls for the development of assessments that target "higher-order thinking," as opposed to mere recall of facts. A common assumption is that this necessitates the use of constructed response or essay-style test questions; however, empirical evidence suggests that this may…

Cognitive validity examines the relationship between what an assessment aims to measure and what it actually elicits from test takers. The present study examined whether multiple-choice items from the National Assessment of Educational Progress (NAEP) grade 12 U.S. history exam elicited the historical thinking processes they were designed to…

In this paper, we discuss the process of identifying and validating students' abilities to think proportionally. More specifically, we describe the methodology we used to identify these proportional reasoning attributes, beginning with the selection and review of relevant literature on proportional reasoning. We then continue with the…

The purpose of this article is to investigate the mathematical practice of proof validation--that is, the act of determining whether an argument constitutes a valid proof. The results of a study with 8 mathematicians are reported. The mathematicians were observed as they read purported mathematical proofs and made judgments about their validity;…

To investigate the cognitive validity of scientific and quantitative reasoning items, "think-alouds" (verbal solutions) were elicited for a general education instrument. Several items were not aligned in terms of anticipated versus actual content, and the instrument's accuracy is questioned. We discuss study weaknesses and merits of this framework…

Analyses of 10 raters' reasoning during think-aloud interviews provided evidence to support a model of the fundamental processes involved in rating standards-based, nonprescriptive portfolios. This process model provides a framework within which to conceptualize sound-rater reasoning and to identify reasoning that distorts the meaning of scores.…

As World Wide Web access expands into schools and homes, children will likely encounter the misinformation often found in this medium. This qualitative study describes strategies employed by sophisticated adult World Wide Web users as they evaluate authentic Web information with the purpose of adapting these strategies for children in K-12…