In this study, we illustrated an application of the confirmatory mixture IRT model for multidimensional tests. We aimed to examine the differences in student performance by domains with a confirmatory mixture IRT modeling approach. A three-dimensional and three-class model was analyzed by assuming content domains as dimensions and cognitive…

The effects of different response option orders on survey responses have been studied extensively. The typical research design involves examining the differences in response characteristics between conditions with the same item stems and response option orders that differ in valence--either incrementally arranged (e.g., strongly disagree to…

Multiple choice results are inherently probabilistic outcomes, as correct responses reflect a combination of knowledge and guessing, while incorrect responses additionally reflect blunder, a confidently committed mistake. To objectively resolve knowledge from responses in an MC test structure, we evaluated probabilistic models that explicitly…

There is a broad literature in multiple-choice test development, both in terms of item-writing guidelines, and psychometric functionality as a measurement tool. However, most of the published literature concerns multiple-choice testing in the context of expert-designed high-stakes standardized assessments, with little attention being paid to the…

This paper presents the results of a confirmatory factor analysis on two self-efficacy scales designed to probe the self-efficacy of college-level introductory astronomy (Astro-101) students (n ¼ 15181) from 22 institutions across the United States of America and Canada. The students undertook a course based on similar curriculum materials, which…

In the Stroop task, congruency effects (i.e., the color-naming latency difference between incongruent stimuli, e.g., the word BLUE written in the color red, and congruent stimuli, e.g., RED in red) are smaller in a list in which incongruent trials are frequent than in a list in which incongruent trials are infrequent. The traditional explanation…

Validity of a measurement refers to appropriate test score meanings, uses, and interpretations (Messick, 1989; Kane, 1992). There are different approaches to validity: an evidentiary aspect of validity is one requiring gathering statistical evidence to evaluate test score meaning. A common approach to validation is comparisons of test score equity…

Increasing linguistic diversity in classrooms has led researchers to examine the validity and fairness of standardized achievement tests, specifically concerning whether test score interpretations are free of bias and score use is fair for all students. This study examined whether mathematics achievement test items that contain complex language…

This article examines whether the way that PISA models item outcomes in mathematics affects the validity of its country rankings. As an alternative to PISA methodology a two-parameter model is applied to PISA mathematics item data from Canada and Finland for the year 2012. In the estimation procedure item difficulty and dispersion parameters are…

According to Dörnyei's model of second language (L2) motivation, the motivated learner aims to incorporate the L2 into his or her self-concept, known as the ideal L2 self. This study examined the internal consistency of Dörnyei's model among ESL Francophone students in Quebec (n = 68) by means of a questionnaire. Correlations were calculated…

This study investigated the effect of cloze item practice on reading comprehension, where cloze items were either created by humans, by machine using natural language processing techniques, or randomly. Participants from Amazon Mechanical Turk (N = 302) took a pre-test, read a text, and took part in one of five conditions, Do-Nothing, Re-Read,…

Item development is a time- and resource-intensive process. Automatic item generation integrates cognitive modeling with computer technology to systematically generate test items. To date, however, items generated using cognitive modeling procedures have received limited use in operational testing situations. As a result, the psychometric…

The authors assessed the effects of using "none of the above" as an option in a 40-item, general-knowledge multiple-choice test administered to undergraduate students. Examinees who selected "none of the above" were given an incentive to write the correct answer to the question posed. Using "none of the above" as the…

A self-report scale that measures teachers' confidence in teaching students about twenty-first century skills was developed and validated with pre-service and in-service teachers. First, 16 items were created to measure teaching confidence in six areas: information literacy, collaboration, communication, innovation and creativity, problem solving,…

A curriculum vitae (CV) is probably the most important first piece of information for an employer to evaluate for job applications (Cole et al. 2007). The CV may lead to an interview for the position advertised. Therefore, writing a good CV is extremely essential and it is a valuable life skill to have. However, there have been debates as to the…