In an Angoff standard setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item in the test. In many cases, these item performance estimates are made twice, with information shared with the panelists between estimates. Especially for long tests, this…

A method of investigating for possible bias in test items is proposed that uses analysis of variance for item data based on groups that have been selected to have identical test score distributions. The item data used are arcsin transformations of item difficulties. The methodological procedure has the following advantages: (1) The arcsin…

A follow-up technique is needed to identify items contributing to items-by-groups interaction when using an ANOVA procedure to examine a test for biased items. The method described includes distribution theory for assessing level of significance and is sensitive to items at all difficulty levels. (Author/GSK)

Two studies examined the effect of making the correct answer of a multiple choice test item grammatically consistent with the item. American College Testing Assessment experimental items were constructed to investigate grammatical compliance to investigate grammatical compliance for plural-singular and vowel-consonant agreement. Results suggest…

The accuracy of standards obtained from judgmental methods is dependent on the quality of the judgments made by experts throughout the standard setting process. One important dimension of the quality of these judgments is the consistency of the judges' perceptions with item performance of minimally competent candidates. Several interrelated…

Three levels of the Iowa Tests of Basic Skills were studied to disclose the possible existence of sex bias in mathematics test items. Two mathematics tests (mathematical concepts and mathematics problem solving) and two comparison verbal tests (vocabulary and reading) were selected for analysis at three levels--grades 3, 6, and 8. Samples of 480…

Analysis of variance and subjective rating by curriculum specialists were used to identify biased items on the Iowa Tests of Basic Skills. Results show little agreement between statistical and subjective methods. Test developers should statistically support a reviewer's selection of biased items. (Author/CP)

The comparability of Angoff-based item ratings on a general education test battery made by judges from within-content and across-content domains was studied. Results with 26 college faculty judges indicate that, at least for some tests, item ratings might be essentially equivalent regardless of judge's content specialty. (SLD)

To determine if equal raw scores have the same "meaning" for students tested "in" and "out-of-level," four "out-of-level" grade groups who took the Vocabulary, Reading, and Mathematics Concepts subtests of the Iowa Tests of Basic Skills typically administered to fifth graders were compared to "in…

An experiment investigated the extent to which the results of out-of-level testing may be biased because the child given an out of level test may have had a significantly different curriculum than the children given in-level tests. Item analysis data suggested this was unlikely. (CTM)

The impact of overall test length and difficulty on the expert judgments of item performance by the Nedelsky method were studied. Five university-level instructors predicting the performance of minimally competent candidates on a mathematics examination were fairly consistent in their assessments regardless of length or difficulty of the test.…

One question regarding the utility of adaptive testing is the effect of individualized item arrangements on examinee test scores. The purpose of this study was to analyze the item difficulty choices by examinees as a function of previous item performance. The examination was a 25-item test of basic algebra skills given to 36 students in an…