Automated scoring of student responses to open-ended questions, including short-answer questions, has great potential to scale to a large number of responses. Recent approaches for automated scoring rely on supervised learning, i.e., training classifiers or fine-tuning language models on a small number of responses with human-provided score…

This research synthesizes studies that used a Digitalized Interactive Component (DIC) to assess K-12 student mathematics performance during Computer-based-Assessments (CBAs) in mathematics. A systematic search identified ten studies that categorized existing DICs according to the tools that provided language assistance to students and tools that…

In online educational systems we can easily collect and analyze extensive data about student learning. Current practice, however, focuses only on some aspects of these data, particularly on correctness of students answers. When a student answers incorrectly, the submitted wrong answer can give us valuable information. We provide an overview of…

Since fall 1976, an undergraduate measurement class has utilized a 32K microcomputer programmed in APL to present replicate sets of questions randomly selected from 80-question item pools representing each of three domains of knowledge based on Sax's Principles of Educational Measurement and Evaluation. Multiple choice questions are presented on a…

Item analysis procedures appropriate for domain-referenced classroom testing are described. A conceptual framework within which item statistics can be considered and promising statistics in light of this framework are presented. The sampling fluctuations of the more promising item statistics for sample sizes comparable to the typical classroom…

The development of the Computer Assisted Testing (CAT) System at the University of Delaware is described. The introduction presents the background leading up to interactive terminal testing in 1973. Documentation for the system includes CAT System programs, format of questions for constructing test item pools, format for entering class lists,…

In connection with the development of item pools for the Navy's experimental Computerized Adaptive Testing (CAT) system, items in five different content areas (General Science, Paragraph Composition, Word Knowledge, Math Knowledge, and Arithmetic Reasoning) have been calibrated using item response theory (IRT) methodology. Operational Armed…

Using Guttman's facet design analysis, four parallel forms of a multiple-choice test were developed. A mapping sentence, logically representing the universe of content of a basic cardiology course, specified the facets of the course and the semantic structural units linking them. The facets were: cognitive processes, disease priority, specific…

This paper reports the results of using a three-layer backpropagation artificial neural network to predict item difficulty in a reading comprehension test. Two network structures were developed, one with and one without a sigmoid function in the output processing unit. The data set, which consisted of a table of coded test items and corresponding…

CSAR (Characteristic Storage And Retrieval) is the name of an interactive item bank system for the storage, retrieval, manipulation, and summarization of information about multiple choice test items. The purpose of the system is to facilitate the test construction process. The system has two main components: one or more item banks and an…

The purpose of this paper is to present two new alternative methods to the current goodness of fit methodology. With the increase use of computerized adaptive test (CAT), the ability to determine the accuracy of calibrated item parameter estimates is paramount. The first method applies a normalizing transformation to the logistic residuals to make…

At Santa Fe Community College, the development of a biology course and departmental final examination was accomplished in the following stages: (1) faculty reached consensus on course content, major objectives and sub-objectives; (2) faculty contributed final examination questions corresponding to the 10 major course topics to a pool for review…

The effects of item position on item statistics were studied in a large set of data from tests of word knowledge (WK) and arithmetic reasoning (AR). Position effects on item response theory (IRT) parameter estimates and classical item statistics were also investigated. Data were collected as part of a project to refine the Army's Computerized…

A computerized flexilevel test was implemented and its ability estimates were compared with those of a Bayesian estimation based computerized adaptive test (CAT) as well as with known true ability estimates. Results showed that when the flexilevel test was terminated according to Lord's criterion, its ability estimates were highly and…

Student achievement test scores were compared and equated, using three different testing methods: paper-administered, computer-administered, and computerized adaptive testing. The tests were developed from third and sixth grade mathematics item banks of the California Assessment Program. The paper and the computer-administered tests were identical…