Highlights test biases that can hamper the performance of adult learners. The biases can originate from the content of the questions or from special situations related to motivation, competition, membership in specific groups, or instructional methods. (JOW)

Criteria for planning, designing, and writing test items are suggested. The criteria were developed via a discussion by subject matter specialists, psychometricians, and test construction experts. Seven criteria proposed for test items of merit address the congruence of an item with its intended purpose, technical assumptions, and editorial…

The absence of the invariant item ordering (IIO) property in two nonparametric polytomous item response theory (IRT) models is discussed, and two nonparametric models are discussed that imply an IIO. Only two parametric polytomous IRT models are found to imply an IIO. A method is proposed to investigate whether an IIO is implied with empirical…

Developed an empirical Bayes enhancement to Mantel-Haenszel (MH) analysis of differential item functioning (DIF) in which it is assumed that the MH statistics are normally distributed and that the prior distribution of underlying DIF parameters is also normal. (Author/SLD)

Investigated empirical Type I error rates and the power of omega (index of answer copying developed by J. Wollack, 1997) when item and trait (theta) parameters were unknown and estimated from datasets of 100 and 500 examinees. Type I error was unaffected by estimating item parameters, with power slightly lower for the smaller sample. (SLD)

Studied whether matching on multiple test scores would reduce false-positive error rates compared to matching on a single number-correct score using simulation. False-positive error rates were reduced for most datasets. Findings suggest that assessing the dimensional structure of a test can be important in analysis of differential item functioning…

Derives a general formula for the population parameter being estimated by the Mantel-Haenszel differential item functioning (DIF) statistics. The formula is appropriate for uniform or nonuniform DIF and can be used regardless of the form of the item response function. Shows the relationship between this parameter and item difficulty. (SLD)

Describes the steps for estimating a population mean from three test items from a test that has been calibrated. Because the Rasch model allows for a direct transformation between raw scores and logit measures, if a population mean in logits is known relative to a set of item calibrations, the population mean in raw score units can be determined.…

Dichotomous item response theory (IRT) models can be viewed as families of stochastically ordered distributions of responses to test items. This paper explores several properties of such distributions, especially those related to transfer to other distributions. Results are formulated as a series of theorems and corollaries that apply to…

Examined intra- and inter-rater consistency of item performance estimated from an Angoff standard setting over 2 years, with 29 panelists one year, and 30 the next. Results provide evidence that item performance estimates were consistent within and across panels within and across years. Factors that might have influenced this high degree of…

Examination of the Affective and Continuance Commitment Scales of J. P. Meyer and N. J. Allen using confirmatory factor analysis for 333 subjects with the LISREL 7 computer program provided strong support across multiple diagnostics for existence of a reverse coding factor defined by the 6 negatively worded scale items. (Author/SLD)

Studied a method for flagging differential item functioning (DIF) based on loss functions. Builds on earlier research that led to the development of an empirical Bayes enhancement to the Mantel-Haenszel DIF analysis. Tested the method through simulation and found its performance better than some commonly used DIF classification systems. (SLD)

Studied two procedures for detecting potentially flawed items in translated tests with small samples: (1) conditional item "p" value comparisons; and (2) delta plots. Varied several factors in this simulation study. Findings show that the two procedures can be valuable in identifying flawed test items, especially when the size of the…

Examined the polytomous differential functioning of items and tests (DFIT) framework proposed by N. Raju and others through simulation. Findings show that the DFIT framework is effective in identifying differential item functioning and differential test functioning. (SLD)

Conducted a regression analysis to assess the contributions of passage and no-passage factors to item variance on the Scholastic Aptitude Test reading comprehension task. Results show that no-passage factors play a larger role than do passage factors, accounting for as much as three-fourths of systematic variance in item difficulty and more than…