As adaptive testing moves toward operational implementation in large scale testing programs, where it is important that adaptive tests be as parallel as possible to existing linear tests, a number of practical issues arise. This paper concerns three such issues. First, optimum item pool size is difficult to determine in advance of pool…

This study evaluated the use of bivariate matching as a solution to the problem of studying differential item functioning (DIF) with formula scored tests. Using Scholastic Aptitude Test verbal data with large samples, both male/female and black/white group comparisons were investigated. Mantel-Haenszel (MH) delta-(D) DIF values and DIF category…

The term differential item functioning (DIF) refers to whether or not the same psychological constructs are measured across different groups. If an item does not measure the same skills or subskills in different populations, it is said to function differentially or to display item bias. A multilevel approach to DIF is proposed. In such a model,…

Several exploratory analyses of the fifths data generated by Test of English as a Foreign Language (TOEFL) item analyses were developed in order to evaluate the effects of options on the discriminability of difficult items and to identify difficult items with low, unreliable biserials that had been rejected by test developers, but for which…

The influence of anchor length on the accuracy of test equating was studied using Tucker's linear method and two Item-Response-Theory (IRT) based methods, focusing on whether equating accuracy improved with more anchor items, whether the anchor effect depended on the equating method used, and the adequacy of the inclusion of the guessing parameter…

The Mantel-Haenszel procedure (N. Mantel and W. Haenszel, 1959) and its extension to constructed response items, the Generalized Mantel Haenszel (A. Agresti, 1990), compare performance of subgroups across different score groups to determine differential item functioning (DIF). At each level of comparison, or score group, the subgroups are…

Analyses of questionnaire data from a program evaluation indicate that the two dichotomous items "Would you recommend this to a friend?" and "Would you choose to do this again?" are not as interchangeable as might be expected from the survey literature. As part of the evaluation of a university program, a survey of graduates…

This study investigated the intrajudge consistency of Angoff-based item performance estimates. The examination used was a certification examination in an emergency medicine specialty. Ten expert panelists rated the same 24 items twice during an operational standard setting study. Results indicate that the panelists were highly consistent, in terms…

A confirmatory approach to assessing test structure using multidimensional item response theory (MIRT) was developed and evaluated. The approach involved adding to the exponent of the MIRT model an item structure matrix that allows the user to specify the ability dimensions measured by an item. Various combinations of item structures were fit to…

Binary or graded disagree-agree responses to attitude items are often collected for the purpose of attitude measurement. Although such data are sometimes analyzed with cumulative measurement models, recent investigations suggest that unfolding models are more appropriate (J. S. Roberts, 1995; W. H. Van Schuur and H. A. L. Kiers, 1994). Advances in…

It is sometimes sensible to think of the fundamental unit of test construction as being larger than an individual item. This unit, dubbed the testlet, must pass muster in the same way that items do. One criterion of a good item is the absence of differential item functioning (DIF). The item must function in the same way as all important…

Among the measurement techniques receiving greater attention is the context-dependent item set or testlet. The context-dependent item set consists of a scenario and related test questions. This item format is generally believed to be able to tap higher level thinking. Unfortunately, this item form leads to inter-item dependence within item sets…

Fit analysis is widely performed in item response theory (IRT) based test development to assess the fit of individual items to the IRT model being used. The paper explores a step fit analysis procedure that is an extension of IRT-based item fit diagnostics applied to the response categories present in popular performance-based tasks. The step fit…

The problem of response set is important for questionnaire designers and interpreters, but the public is affected as well if policy is determined on the basis of unsupported conclusions. This study focused on one of the most researched response sets, extreme responding (ER), or extreme checking styles, and its relationship to one dimension of…

An item-selection algorithm to neutralize the differential effects of time limits on scores on computerized adaptive tests is proposed. The method is based on a statistical model for the response-time distributions of the examinees on items in the pool that is updated each time a new item has been administered. Predictions from the model are used…