Performance-based assessment is a common approach to assess the development and acquisition of practice competencies among health professions students. Judgments related to the quality of performance are typically operationalized as ratings against success criteria specified within a rubric. The extent to which the rubric is understood,…

In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes. Learners are introduced to historical, theoretical, and procedural differences between these methods and how to use and analyze…

This article illustrates and differentiates the unique role cognitive interviews and think-aloud interviews play in developing and validating assessments. Specifically, we describe the use of (a) cognitive interviews to gather empirical evidence to support claims about the intended construct being measured and (b) think-aloud interviews to gather…

Cognitive interviewing (CI) provides a method of systematically collecting validity evidence of response processes for questionnaire items. CI involves a range of techniques for prompting individuals to verbalise their responses to items. One such technique is concurrent verbalisation, as developed in Think Aloud Protocol (TAP). This article…

Over the last three decades, there has been increased attention on the collection and interpretation of "response processing data" to inform claims of learners' knowledge and skills (e.g., see Ercikan et al., 2010; Kobrin & Young, 2003; see also, Leighton, 2004). Response processing data are perhaps most consequential in the…

The purpose of this article is to describe the process of a proof construction. It is more specific on the failure of the process. Piaget's frameworks, assimilation and accommodation, were used to analyze it. Method of this research was qualitative method. Data were collected by asking five students working on problems of proof using think aloud…

Instruction promoting reasoning and proof should be a part of K-12 mathematics teaching. Preservice teachers (PSTs) need mathematics content instruction that helps them construct valid justifications for mathematical notions and develop positive perceptions of proof. This case study describes instruction with proof-related tasks and manipulatives…

The paper aims to demonstrate the value of cognitive interviewing (CI) as a survey pretesting method in comparative education research. Although rarely used by education researchers, CI has been successfully applied in different disciplines to evaluate and improve question performance. The method assumes that observing people's thought processes…

In this paper, we discuss the process of identifying and validating students' abilities to think proportionally. More specifically, we describe the methodology we used to identify these proportional reasoning attributes, beginning with the selection and review of relevant literature on proportional reasoning. We then continue with the…

The goal of this study was to investigate the usefulness of person-fit analysis in validating student score inferences in a cognitive diagnostic assessment. In this study, a two-stage procedure was used to evaluate person fit for a diagnostic test in the domain of statistical hypothesis testing. In the first stage, the person-fit statistic, the…

Interest in student conception of experimentation inspired the development of a fully validated 14-question inventory on experimental design in biology (BEDCI) by following established best practices in concept inventory (CI) design. This CI can be used to diagnose specific examples of non-expert-like thinking in students and to evaluate the…

The Standards for Educational and Psychological Testing indicate that multiple sources of validity evidence should be used to support the interpretation of test scores. In the past decade, examinee response processes, as a source of validity evidence, have received increased attention. However, there have been relatively few methodological studies…

This paper provides an example of the application of the cognitive interview, a qualitative tool for pre-testing a survey instrument to check its cognitive validity, that is, whether the items mean to respondents what they mean to the item designers. The instrument is the survey used in the final phase of the International Study of Principal…

Glossary and reading aloud test items are commonly allowed in many states' accommodation policies for English language learner (ELL) students for large-scale mathematics assessments. However, little research is available regarding the effects of these accommodations on ELL students' performance. Further, no research exists that examines how…

Present conceptualizations and measures of self-esteem do not account for linguistic self-esteem, an aspect of the self specifically relevant for bilingual students. This study examines the utility of a newly developed measure of linguistic self-esteem. This novel measure is compared with a commonly used self-esteem measure, two standardized…