Background: Both learning and problem solving are major goals of complex problem solving in engineering education. The order of knowledge construction and problem solving in learning through problem solving, however, has not been explained in current literature. Objectives: To understand their relationships, this study compared the effects of…

This article illustrates and differentiates the unique role cognitive interviews and think-aloud interviews play in developing and validating assessments. Specifically, we describe the use of (a) cognitive interviews to gather empirical evidence to support claims about the intended construct being measured and (b) think-aloud interviews to gather…

This paper marks the second in a series of five reports detailing the work of the Optimizing Assessment for All (OAA) project at Brookings to strengthen education systems capacity to integrate 21st century skills into teaching and learning, using assessment as a lever for changing classroom practices. Twenty-first century skills (21CS) are now…

The important but little understood problem that motivated this study was the lack of research on valid assessment methods to determine progress in higher-order learning in situations involving complex and ill-structured problems. Without a valid assessment method, little progress can occur in instructional design research with regard to designing…

This preliminary study described fourth grade students verbally solving a mathematics problem using a think-aloud protocol. Comments in the think aloud were categorized according to type (e.g., paraphrases and elaborations) and facilitative nature (i.e., whether the comments facilitated correct solution of the problem). Amount of the students'…

Age-related changes in the use of verbal processes for the efficient switching between tasks were investigated in 5-year-old children (N = 32, M age = 5.9 years) and 9-year-old children (N = 32, M age = 9.4 years). Task switching was assessed by means of a cued switching paradigm to examine two switching components: (a) to maintain and select and…

Describes differences in formative evaluation techniques between instructional designers and subject matter experts. Content analysis based on think-aloud data focusing on differences in task interpretation, problem identification and revision statements, and review strategies is discussed; use of instructional systems design models by…

This report characterizes human problem solving in digital circuit design. Protocols of 11 different designers with varying degrees of training were analyzed by identifying the designers' problem solving strategies and discussing activity patterns that differentiate the designers. These methods are proposed as a tentative basis for assessing…

Thirty-one students (ages 8-14) with learning disabilities or behavior disorders solved arithmetic word problems while a narrative protocol was developed. Eight types of cognitive processing difficulties were identified (such as lack of self-correction and inclusion of extraneous information), and the utility of protocol analysis as an assessment…

This study examined planning processes and the acquisition of expert problem solving skills among individuals of different expertise on tasks of increasing complexity. Twelve French-speaking adult students of machining (novices) and 12 French-speaking teachers of machining (experts) in Quebec (Canada) were randomly assigned to a simple or complex…

Whether high levels of metacognitive knowledge about problem solving can compensate for low overall aptitude was studied for 31 high-aptitude and 25 low-aptitude children in grades 4 and 5. Results confirming superior performance of high metacognitive individuals are discussed in terms of the independence of metacognition and academic aptitude.…

Proposes the use of procedural task analysis as a tool to assess a person's mental processes during problem solving in science education research. Uses transcriptions of videotape or audiotape problem-solving sessions to identify students' knowledge structures and misconceptions. (MDH)

How people evaluate primary and secondary sources when considering historical questions was studied by asking eight historians and eight high school seniors to think aloud while reviewing written and pictorial documents about the Battle of Lexington. Differences between the groups are discussed concerning cognitive processes and beliefs about…