Response process validity evidence provides a window into a respondent's cognitive processing. The purpose of this study is to describe a new data collection tool called a whole-class think aloud (WCTA). This work is performed as part of test development for a series of problem-solving measures to be used in elementary and middle grades. Data from…

The use of computer-based tests (CBTs), for both formative and summative purposes, has greatly increased over the past years. One major advantage of CBTs is the easy integration of multimedia. It is unclear, though, how to design such CBT environments with multimedia. The purpose of the current study was to examine whether guidelines for designing…

Although an accurate measure of vocabulary size is integral to understanding the proficiency of language learners, the validity of multiple-choice (M/C) vocabulary tests to determine this has been questioned due to users guessing correct answers which inflates scores. In this paper the nature of guessing and partial knowledge used when taking the…

In line with the widespread proliferation of digital technology in everyday life, many countries are now beginning to use computer-based exams (CBEs) in their post-primary education systems. To ensure that these CBEs are delivered in a manner that preserves their fairness, validity, utility and credibility, several factors pertaining to their…

In this study, we describe the methodology used to identify and validate a set of expert-defined fraction subtraction related attributes. These attributes are expected to be mastered by 6th grade students toward proficiency in fraction subtraction. This research argues and demonstrates that state standards guiding subject instruction plays an…

Across education, certification and licensure, there are repeated calls for the development of assessments that target "higher-order thinking," as opposed to mere recall of facts. A common assumption is that this necessitates the use of constructed response or essay-style test questions; however, empirical evidence suggests that this may…

This study compared an expert-based cognitive model of domain mastery with student-based cognitive models of task performance for Integrated Algebra. Interpretations of student test results are limited by experts' hypotheses of how students interact with the items. In reality, the cognitive processes that students use to solve each item may be…

This article considers the design process for new formative history assessments. Over the course of 3 years, my colleagues from the Stanford History Education Group and I designed, piloted, and revised dozens of "History Assessments of Thinking" (HATs). As we created HATs, we sought to gather information about their cognitive validity,…

Findings derived from self-reported, structured survey questionnaires are commonly used in evaluation and applied research to inform policy-making and program decisions. Although there are a variety of issues related to the quality of survey evidence (e.g., sampling precision), the validity of response processes--how respondents process thoughts…

Clear descriptions of job expertise are required to support applications and improvements in personnel training and job performance. This report describes a practical approach to task analysis that integrates the issues, content, and methods of cognitive science and personnel psychology. Cognitively oriented task analysis employs a breadth, then…

A study examined whether the process of gathering verbal reports of subjects' thinking while taking multiple-choice critical thinking tests could be used to infer the reasoning process used and identify test items which do not require critical thinking skills. Four factors can render an inference of a subject's critical thinking skills…

The domination of the information processing approach has shifted research from problem solving strategies to the structure and organization of knowledge that characterizes expertise. The purpose of this study was to compare the reasoning processes of 12 clinicians and 40 medical students as they responded to 6 positively stated multiple choice…

This paper describes the task analysis of performance-based science tasks that were designed for the 1994 National Assessment of Educational Progress (NAEP) science assessment, now postponed until 1996, and field tested in 1993. A brief description of the science performance tasks is followed by a description of the task analyses performed and a…

Many current efforts to develop large-scale science assessments involve hands-on tasks because of their presumed power to elicit and measure scientific reasoning skills. An analysis of the processes in which students engage while responding to such assessment is needed in order to discover the specific forms of reasoning that tasks elicit. This…

Analyses of the cognitive activity displayed by students on existing innovative assessments in science were conducted. The project has worked with pilot alternative assessment programs in Connecticut and California to document the match or mismatch between the skills and processes that the assessment is designed to tap and those actually elicited.…