Multiple choice questions are traditionally expensive to produce. Recent advances in large language models (LLMs) have led to fine-tuned LLMs that generate questions competitive with human-authored questions. However, the relative capabilities of ChatGPT-family models have not yet been established for this task. We present a carefully-controlled…

The multiple-choice (MC) item format has been widely used in educational assessments across diverse content domains. MC items purportedly allow for collecting richer diagnostic information. The effectiveness and economy of administering MC items may have further contributed to their popularity not just in educational assessment. The MC item format…

Exams constitute the predominant form of summative assessment in undergraduate biology education, with the assumption that exam performance should reflect student conceptual understanding. Previous work highlights multiple examples in which students can answer exam problems correctly without the corresponding conceptual understanding. This…

An algorithm for item analysis in which item discrimination indices have been defined for the distractors as well as the correct answer is presented. Also, the concept of convergent and discriminant validity is applied to items instead of tests, and is discussed as an aid to item analysis. (Author/JKS)

Multiple-choice tests are generally scored using a conventional number right scoring method. While this method is easy to use, it has several weaknesses. These weaknesses include decreased validity due to guessing and failure to credit partial knowledge. In an attempt to address these weaknesses, psychometricians have developed various scoring…

This study is a critical evaluation of the roles for coding and scoring of missing responses to multiple-choice items in educational tests. The focus is on tests in which the test-takers have little or no motivation; in such tests omitting and not reaching (as classified by the currently adopted operational rules) is quite frequent. Data from the…

A latent class model for handling the items in Birenbaum and Tatsuoka's study is described. A method to derive the optimal scoring rule when multiple choice test items are used is illustrated. Remedial training begins after a determination is made as to which of several erroneous algorithms is being used. (Author/DWH)

Twenty-four students who scored 650 or more on the Scholastic Aptitude Test Mathematics test (SAT-M) were asked to think aloud while solving 13 mathematics items in either multiple-choice or free-response format. Strategies students used to solve the items were classified as either algorithmic or insightful. Data analyses indicated that items in…

In this report, selection theory is used as a theoretical framework from which mathematical algorithms for tailored testing are derived. The process of tailored, or adaptive, testing is presented as analogous to personnel selection and rejection on a series of continuous variables that are related to ability. Proceeding from a single common-factor…

The aim of this study was to show that the superiority of corrected-for-guessing scores over number right scores as true score estimates depends on the ability of examinees to recognize situations where they can eliminate one or more alternatives as incorrect and to omit items where they would only be guessing randomly. Previous investigations…

The study was designed to examine the similarity of response options generated by different item writers using a systematic approach to item writing. The similarity of response options to student responses for the same item stems presented in an open-ended format was also examined. A non-systematic (subject matter expertise) approach and a…

On well-constructed multiple-choice tests, the most serious threat to measurement is not variation in item discrimination, but the guessing behavior that may be adopted by some students. Ways of ameliorating the effects of guessing are discussed, especially for problems in latent trait models. A new item response model, including an item parameter…

An earlier study was extended and replicated to examine the feasibility of generating multiple-choice test questions by transforming sentences from prose instructional material. In the first study, a computer-based algorithm was used to analyze prose subject matter and to identify high-information words. Sentences containing selected words were…

The results of this study are consistent with a two-stage model of learning chemistry, a multi-dimensional subject, in which students accumulate knowledge in stage one, and then restructure their knowledge in stage two. When cognitive, metacognitive and achievement variables were subjected to a predictive discriminant analysis (PDA) procedure,…

The feasibility of generating multiple-choice test questions by transforming sentences from prose instructional materials was examined. A computer-based algorithm was used to analyze prose subject matter and to identify high-information words. Sentences containing selected words were then transformed into multiple-choice items by four writers who…