Several issues relating to agreement estimates for different types of data from performance evaluations are considered. New indices of agreement are presented for ordinal level items and for summative scores produced by nominal or ordinal level items. Two sets of empirical data illustrate the performance of the two formulas derived to estimate…

Language testing is an area of applied linguistics that combines the exercise of professional judgment about language, learning, and the nature of the achievement of language learning with empirical data about student performance and, by inference, their abilities. The relationship between judgments and empirical data in language testing is…

The Mantel-Haenszel (MH) procedure attempts to identify and quantify differential item performance (item bias). This paper summarizes the MH statistics, and identifies the parameters they estimate. An equivalent procedure based on the Rasch model is described. The theoretical properties of the two approaches are compared and shown to require the…

This paper discusses the development and use of "item shells" in constructing multiple-choice tests. An item shell is a "hollow" item that contains the syntactic structure and context of an item without specific content. Item shells are empirically developed from successfully used items selected from an existing item pool. Use…

An item discriminability study of the Iowa Tests of Basic Skills Language Skills tests identified test items that are robust discriminators, psychometrically capable of separating low scorers from higher scorers in the language tests battery. The analysis was conducted by calculating a point-biserial correlation for each item on the four language…

A study of 696 undergraduates at the University of Northern Colorado was undertaken to determine the effects of computerized unit test item feedback on final examination scores. The study, which employed the PHOENIX computer managed instruction system, included students at all undergraduate levels enrolled in an Oceanography course. To determine…

The definition of differential item performance (DIP), often referred to as item bias, is discussed. DIP is suggested as a comprehensive term to encompass item bias (item invalidity which is unfair to certain population subgroups) and instructional bias (a valid reflection of group differences in instruction or background). This study investigated…

A new approach to assessing unexpected differential item performance (item bias or item fairness) is introduced and applied to the item responses of different subpopulations of Scholastic Aptitude Test (SAT) takers. The essential features of the standardization approach are described. The primary goal of the standardization approach is to control…

In this practical, easy-to-use guide, suggestions are made for setting educational objectives and preparing a teacher-made test. Specific suggestions are made for selecting items for essay tests, multiple choice tests, completion tests, matching tests, and true-false tests. (BW)

One form of adaptive testing involves a two-stage procedure. The first stage is the administration of a routing test. From this first test, an estimate of an examinee's ability is obtained. On the basis of this ability estimate, a second test focused on a given ability level is administered. The purpose of this study was to compare the efficiency…

The purpose of this released exercise set is to provide easy access to released exercises from the National Assessment of Educational Progress (NAEP) third mathematics assessment, conducted in 1981-82. Documentation includes basic reference numbers, objective classifications National Assessment has found useful, timing and administration data for…

The difficulty of test items administered by paper and pencil were compared with the difficulty of the same items administered by computer. The study was conducted to determine if an interaction exists between mode of test administration and ability. An arithmetic reasoning test was constructed for this study. All examinees had taken the Armed…

The purpose of this paper is to further the cause of clarifying construct interpretations of tests, by proposing that non-metric multidimensional scaling may be more useful than factor analysis or other latent structure models for investigating the internal structure of tests. It also suggests that typical problems associated with scaling…

The purpose of this monograph is twofold: (1) to describe the advantages and disadvantages of mail questionnaires as survey research instruments, and (2) to provide guidelines for constructing and utilizing mail questionnaires with a view to maximizing response rates and minimizing response biases. Construction of an effective mail questionnaire…

A new item type was developed, incorporating features of "ill-structured" problems in a multiple-choice format. The problems are similar to previously developed scientific thinking tasks in requiring the examinee to go beyond the information provided; they resemble a variant of the logical reasoning item type, but demand somewhat more structuring…