There has been increasing attention to the fine structure of abilities underlying task performance (E. H. Haertel and D. E. Wiley). One of the most useful approaches for this is by use of graphical models to represent relationships among abilities and test items. Building a large graphical model is always an issue. Restrictions upon experiments…

Some of the test questions for the National Council of Architectural Registration Boards deal with the site, including drainage, regrading, and the like. Some questions are most easily scored by examining contours, but others, such as water flow questions, are best scored from a grid in which each element is assigned its average elevation. This…

Loglinear and logit models that have been suggested for studying differential item functioning (DIF) are reviewed, and loglinear formulations of the logit models are given. A polynomial loglinear model for assessing DIF is introduced that incorporates scores on the matching variable and item responses. The polynomial loglinear model contains far…

Basic issues in understanding Item Response Theory (IRT), or Latent Trait Theory, measurement models are discussed. These theories have gained popularity because of their promise to provide greater precision and control in measurement involving both achievement and attitude instruments. IRT models implement probabilistic techniques that yield…

Item and test analyses can be used to revise and improve both test items and the test as a whole. Recommendations for item and test analysis practices as they are reported in commonly used measurement textbooks are summarized. A heuristic data set is used to illustrate test and item analysis practices. Techniques developed in this paper are…

This discussion of new methods for calibrating item response theory (IRT) models looks into new optimization procedures, such as the Genetic Algorithm (GA) to improve on the use of the Newton-Raphson procedure. The advantages of using a global optimization procedure like GA is that this kind of procedure is not easily affected by local optima and…

Six methods for assembling tests from a pool with an item-set structure are presented. All methods are computational and based on the technique of mixed integer programming. The methods are evaluated using such criteria as the feasibility of their linear programming problems and their expected solution times. The methods are illustrated for two…

This study derived an expectation-maximization (EM) algorithm for estimating the parameters of multidimensional item response models. A genetic algorithm (GA) was developed to be used in the maximization step in each EM cycle. The focus of the EM-GA algorithm developed in this paper was on multidimensional items with "mixed structure."…

The development of a new technique, the Relative Difficulty Ratio (RDR), is described, as well as how it can be used to determine the difficulty level of a test so that meaningful inter-test difficulty comparisons can be made. Assumptions made in computing RDR include: 1) each item must be scored dichotomously with only one answer choice keyed as…

The G index of agreement does not permit the use of various weights for its various items. The weighted G index described here, make it possible to use unequal weights. An example of the procedure is provided. (Author/JKS)

A statistical experiment was conducted in an operational setting to determine the contributions of different sources of variability to the unreliability scoring of essays and other free-response questions. Partially balanced incomplete block designs facilitated the unbiased estimation of certain main effects without requiring readers to assess the…

The author discusses writing items for a multiple-choice test. Topics include (1) formatting, (2) central theme development, (3) stem revision, (4) distractors, and (5) test validity and reliability. (CH)

Some problems associated with the criteria of reproducibility and scalability as they are used in Guttman scalogram analysis to evaluate cumulative, nonparametric scales of dichotomous items are discussed. A computer program is presented which analyzes response patterns elicited by dichotomous scales designed to be cumulative. (Author/DWH)

The purpose of this study was to investigate and to quantify the tolerable error in item parameter estimates for different sets of items used in computer-based testing. The study examined items that were administered repeatedly to different examinee samples over time, examining items that were administered linearly in a fixed order each time they…

In the Angoff standard setting procedure, subject matter experts (SMEs) estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item comprising the test. In many cases, these item performance estimates are made twice, with information shared with SMEs between estimates. This…