The intricate relationship between representational competence and content knowledge in organic chemistry has been widely debated, and the ways in which representations contribute to task difficulty, particularly in assessment, remain unclear. This paper presents a multiple-choice test instrument for assessing individuals' knowledge of fundamental…

Answering the call for response process-based validation, this study shows how researchers can evaluate validity evidence of test scores based on the mental "processes" test-takers use, rather than based on correlations with other "outcome" measures. The proposed methods for process-based validation studies are demonstrated…

Eye-tracking technology can create a record of the location and duration of visual fixations as a test-taker reads test questions. Although the cognitive process the test-taker is using cannot be directly observed, eye-tracking data can support inferences about these unobserved cognitive processes. This type of information has the potential to…

Although an accurate measure of vocabulary size is integral to understanding the proficiency of language learners, the validity of multiple-choice (M/C) vocabulary tests to determine this has been questioned due to users guessing correct answers which inflates scores. In this paper the nature of guessing and partial knowledge used when taking the…

This article describes the results of Action Research conducted in an ESP classroom of Dhofar University located in Oman. Following the call of Oman Vision 2040 to emphasize educational practices that promote the development of higher-order cognitive processes, this study raises the following question: Can an online multiple choice question (MCQ)…

Investigating the processes underlying test performance is a major source of data supporting the explanation inference in the validity argument (Chappelle, 2021). One way of modeling the cognitive processes underlying test performance is by constructing a Q-matrix, which is essentially about summarizing the attributes explaining test-takers'…

In general, researchers attempt to quantify cognitive load using physiologic and psychometric measures. Although the construct measured by both of these metrics is thought to represent overall cognitive load, there is a paucity of studies that compares these techniques to one another. The authors compared data obtained from one physiologic tool…

This study evaluates a 12-item instrument for subjective measurement of mental load (ML) and mental effort (ME) by analysing different sources of validity evidence. The findings of an expert judgement (N = 8) provide "evidence based on test content" that the formulation of the items corresponds to the meaning of ML and ME. An empirical…

Multiple-choice questions are an important part of large-scale summative assessments, such as the advanced placement (AP) chemistry exam. However, past AP chemistry exam items often lacked the ability to test conceptual understanding and higher-order cognitive skills. The redesigned AP chemistry exam shows a distinctive shift in item types toward…

Standard unidimensional Rasch models assume that persons with the same ability parameters are comparable. That is, the same interpretation applies to persons with identical ability estimates as regards the underlying mental processes triggered by the test. However, research in cognitive psychology shows that persons at the same trait level may…

"Item difficulty modelling" is a technique used for a number of purposes such as to support future item development, to explore validity in relation to the constructs that influence difficulty and to predict the difficulty of items. This research attempted to explore the factors influencing question difficulty in a general qualification…

This document reviews studies that investigated certain inadequacies of English as a Second Language (ESL) reading comprehension tests. The first study examined whether three ESL reading comprehension tests required information principally from the text or from background knowledge. Results indicated that background knowledge differentially…

A study examined the effect of verbal reporting of students' thinking on their performance during an examination. Subjects, 343 high school seniors, were randomly divided into 4 experimental groups, and a different procedure for eliciting students' thinking during a critical thinking test was used for each group. A control group took the same test…

A study examined whether the process of gathering verbal reports of subjects' thinking while taking multiple-choice critical thinking tests could be used to infer the reasoning process used and identify test items which do not require critical thinking skills. Four factors can render an inference of a subject's critical thinking skills…

Preliminary psychometric data assessing the reliability and validity of a method used to measure the diagnostic reasoning and problem-solving skills of predoctoral students in orthodontia are described. The measurement approach consisted of sets of patient demographic data and dental photos and x-rays, accompanied by a set of 33 multiple-choice…