It is imperative to collect validity evidence prior to interpreting and using test scores. During the process of collecting validity evidence, test developers should consider whether test scores are contaminated by sources of extraneous information. This is referred to as construct irrelevant variance, or the "degree to which test scores are…

A conceptualization of multiple-choice exams in terms of signal detection theory (SDT) leads to simple measures of item difficulty and item discrimination that are closely related to, but also distinct from, those used in classical item analysis (CIA). The theory defines a "true split," depending on whether or not examinees know an item,…

As implementation of the "Next Generation Science Standards" moves forward, there is a need for new assessments that can measure students' integrated three-dimensional science learning. The National Research Council has suggested that these assessments be multicomponent tasks that utilize a combination of item formats including…

There is a broad literature in multiple-choice test development, both in terms of item-writing guidelines, and psychometric functionality as a measurement tool. However, most of the published literature concerns multiple-choice testing in the context of expert-designed high-stakes standardized assessments, with little attention being paid to the…

HOTS is one part of the skills that need to be developed in the 21st Century . This study aims to determine the characteristics of the Fundamental Physics Higher-order Thinking Skill (FundPhysHOTS) test for prospective physics teachers using Item Response Theory (IRT) analysis. This study uses a quantitative approach. 254 prospective physics…

In this study, we investigated whether the use of smartphone games while reading a text reduces learning performance or reading speed. We also examined whether this is affected by push notifications. Ninety-three students were randomly assigned to three learning conditions. In the gaming group (G), participants played a game app for 20 s at 2-min…

We developed a Quantum Mechanics Conceptual Understanding Survey (QMCUS) in this study. The survey was conducted using a quantitative methodology. A multiple-choice survey of 35 questions was administered to 338 undergraduate students. Three experienced quantum mechanics instructors examined the validity of the survey. The reliability of our…

This research aimed to investigate animation-based achievement tests according to the item format, psychometric features, students' performance, and gender. The study sample consisted of 52 fifth-grade students in Samsun/Turkey in 2017-2018. Measures of the research were open-ended (OE), animation-based open-ended (AOE), multiple-choice (MC), and…

For a multiple-choice test under development or redesign, it is important to choose the optimal number of options per item so that the test possesses the desired psychometric properties. On the basis of available data for a multiple-choice assessment with 8 options, we evaluated the effects of changing the number of options on test properties…

Many university programs include Microsoft Excel courses given their value as a scientific and technical tool. However, evaluating what is effectively learned by students is a challenging task. Considering multiple-choice written exams are a standard evaluation format, this study aimed to uncover the features influencing students' success in…

Each year students spend thousands of hours participating in discussion forums. Yet there has not been sufficient scholarship on the observable effects of discussion forums on learning outcomes, and there is not a clear understanding of the role discussion forums play in transmitting knowledge of course concepts to students. This study…

This classroom experiment investigates the effects of adding representational pictures to multiple-choice and constructed-response test items to understand the role of the response format for the multimedia effect in testing. Participants were 575 fifth- and sixth-graders who answered 28 science test items--seven items in each of four experimental…

This study investigates the effects of adding stem preview to an English for Academic Purposes (EAP) multiple-choice listening assessment. In stem preview, listeners may view the item stems, but not response options, before listening. Previous research indicates that adding preview to an exam typically decreases difficulty, but raises concerns…

The present study investigates the best factor for controlling the item difficulty of multiple-choice English vocabulary questions generated by an automatic question generation system. Three factors are considered for controlling item difficulty: (1) reading passage difficulty, (2) semantic similarity between the correct answer and distractors,…

The Discrete Option Multiple Choice (DOMC) item format was introduced by Foster and Miller (2009) with the intent of improving the security of test content. However, by changing the amount and order of the content presented, the test taking experience varies by test taker, thereby introducing potential fairness issues. In this paper we…