A method of estimating Rasch-model difficulty calibrations from judges' ratings of item difficulty is described. The ability of judges to estimate item difficulty was assessed by correlating estimated and empirical calibrations on each of four examinations offered by the American Association of State Social Work Boards. Thirteen members of the…

Differential item functioning (DIF), or item bias, occurs when individuals in a focal group respond differently to a test item than do individuals in a reference group even when comparisons are restricted to individuals with similar overall skill levels on the trait in question. It is common in constructing a questionnaire or survey to recommend…

The French as a Second Language model tests for advanced levels 7, 8, and 9 were designed to evaluate students' language performance, as outlined in the program of studies for Alberta, Canada, in listening and reading comprehension and oral and written production, communication skills, culture, language and general language knowledge. The tests…

Using an anchor-item design of test equating, the effects of three equating methods (Tucker linear and two three-parameter item-response-theory-based (3PL-IRT) methods), and the content representativeness of anchor items on the accuracy of equating were examined; and an innovative way of evaluating equating accuracy appropriate for the particular…

This study was conducted to investigate the construct validity of scores on the Personal Preferences Self-Description Questionnaire (PPSDQ), a measure of Jungian types. Confirmatory factor analysis methods were used to investigate the structures underlying PPSDQ responses of 641 university students. The model fit statistics were generally…

This paper presents six steps in test construction generally recommended by measurement textbook authors. The focus is primarily on paper-and-pencil achievement tests as used by class instructions, although the discussion touches on the construction of other types of assessment. The six steps are: (1) determine the test purpose; (2) determine the…

In previous works, D. J. Woodruff derived expressions for three different conditional test score variances: (1) the conditional standard error of prediction (CSEP); (2) the conditional standard error of measurement in prediction (CSEMP); and (3) the conditional standard error of estimation (CSEE). He also presented step-up formulas that require…

This study investigated the impact of task directions on the mathematical performance of high school students from six classes. Students analyzed data regarding school dropout by answering six short-answer questions and writing a letter discussing the trends and their predictions about school dropout. Tasks were scored using two methods: (1) trait…

Fit statistics provide a direct measure of assessment accuracy by analyzing the fit of measurement models to an individual's (or group's) response pattern. Students that lose interest during the assessment, for example, will miss exercises that are within their abilities. Such students will respond correctly to some more difficult items and…

This paper reports a method for assessing the adequacy of existing item banks for computer adaptive testing. The method takes into account content specifications, test length, and stopping rules, and can be used to determine if an existing item bank is adequate to administer a computer adaptive test efficiently across differing levels of examinee…

A method is presented for the simultaneous analysis of differential item functioning (DIF) in multi-factor situations. The method is unique in that it combines item response theory (IRT) and analysis of variance (ANOVA), takes a simultaneous approach to multifactor DIF analysis, and is capable of capturing interaction and controlling for possible…

The purposes of this study were to introduce the iterative purification procedure and to compare this with the two-step purification procedure, to compare false positive error rates and the power of five observed score approaches and to identify factors affecting power and false positive rates in each method. This study used 2,400 data sets that…

Accuracy of item parameter estimates is a critical concern for any application of item response theory (IRT). However, the necessary sample sizes are often difficult to obtain in practice, particularly for the more complex models. A promising avenue of research concerns modified item response models. This study both replicates and improves on an…

Several person-fit statistics have been proposed to detect item score patterns that do not fit an item response theory model. To classify response patterns as not fitting a model, a distribution of a person-fit statistic is needed. The null distributions of several fit statistics have been investigated using conventionally administered tests, but…

This report examines both the substantiveness and statistical significance of differences between mean American College Testing (ACT) Program assessment scale scores. The first part of the report describes the development and results of a method for interpreting the substantiveness of mean differences in terms of test items correctly answered. For…