Many researchers use Cronbach's alpha to demonstrate internal consistency, even though it has been shown numerous times that Cronbach's alpha is not suitable for this. Because the intention of questionnaire and test constructers is to summarize the test by its overall sum score, we advocate summability, which we define as the proportion of total…

In typical differential item functioning (DIF) assessments, an item's DIF status is not influenced by its status in previous test administrations. An item that has shown DIF at multiple administrations may be treated the same way as an item that has shown DIF in only the most recent administration. Therefore, much useful information about the…

In most contexts of student skills assessment, whether the test material is administered by the teacher or within a learning environment, there is a strong incentive to minimize the number of questions or exercises administered in order to get an accurate assessment. This minimization objective can be framed as a Q-matrix design problem: given a…

We ask questions of students in order to probe their understanding. We design our questions in such a way that we can assess a student's progress towards an accurate worldview. However, there is a consensus that a performance gap exists in many physics assessments, where male students outperform their female peers. While early work in this area…

Evolutionary trees are central to learning about evolutionary processes, yet students at all educational levels struggle to read and interpret them. The synthetic tree-reading model (STREAM), based on published and not yet empirically tested models, was tested to determine whether the assumed hierarchy of the model could be substantiated and how…

In contrast with the assumptions made in standard measurement models used in large-scale assessments, students' performance may change during the test administration. This change can be modeled as a function of item position in case of a test booklet design with item-order manipulations. The present study used an explanatory item response theory…

This study presents results from a randomized experiment in the 2015-2017 National Survey of Family Growth, where a large national sample of U.S. individuals aged 15-49 was randomly assigned to one of two different versions of a survey question about sexual identity (one with three response options, including heterosexual, gay/lesbian, and…

Effectively evaluating the credibility and accuracy of multiple sources is critical for college readiness. We developed 24 source evaluation tasks spanning four predicted difficulty levels of a hypothesized learning progression (LP) and piloted these tasks to evaluate the utility of an LP-based approach to designing formative literacy assessments.…

There is a broad literature in multiple-choice test development, both in terms of item-writing guidelines, and psychometric functionality as a measurement tool. However, most of the published literature concerns multiple-choice testing in the context of expert-designed high-stakes standardized assessments, with little attention being paid to the…

The Neurodevelopmental Ecological Screening Tool (NEST) is a new instrument to screen children for developmental challenges. This article describes the validation of the NEST neurodevelopmental domain. Data were collected from a nationwide purposely restricted sample of caregivers of children aged 3-5 years (n = 231) living in poverty and…

University lecturers and coordinators of business and economics courses around the world are faced with the challenge that beginning students in these courses have heterogeneous entry conditions in terms of personal characteristics. This article focuses on the economic knowledge of German and Japanese beginning students in a business and economics…

In a signal detection theory (SDT) approach to multiple choice exams, examinees are viewed as choosing, for each item, the alternative that is perceived as being the most plausible, with perceived plausibility depending in part on whether or not an item is known. The SDT model is a process model and provides measures of item difficulty, item…

In Programme for International Student Assessment (PISA), item response theory (IRT) scaling is used to examine the psychometric properties of items and scales and to provide comparable test scores across participating countries and over time. To balance the comparability of IRT item parameter estimations across countries with the best possible…

The purpose of this paper is to briefly introduce two most common applications of multiple group item response theory (IRT) models, namely detecting differential item functioning (DIF) analysis and nonequivalent group score linking with a simultaneous calibration. We illustrate how to conduct those analyses using the "Stata" item…

This research examines the difficulty pattern of assertive communication scale instrument items containing spiritual values. The research and development design applies ADDIE work procedures (Analysis, Design, Development or Production, Implementation or delivery and Evaluation). The participants of the item development and item difficulty test…