This technical report and user's guide is designed to provide researchers with an overview of the design and implementation of the 2009 Program for International Student Assessment (PISA), as well as with information on how to access the PISA 2009 data. This information is meant to supplement that presented in Organization for Economic Cooperation…

This paper discusses loglinear models for assessing differential item functioning (DIF). Loglinear and logit models that have been suggested for studying DIF are reviewed, and loglinear formulations of the logit models are given. A polynomial loglinear model for assessing DIF is introduced. Two examples using the polynomial loglinear model for…

The item response function (IRF) for a polytomously scored item is defined as a weighted sum of the item category response functions (ICRF, the probability of getting a particular score for a randomly sampled examinee of ability theta). This paper establishes the correspondence between an IRF and a unique set of ICRFs for two of the most commonly…

The one observation per cell two-way items by examinees random effects analysis of variance (ANOVA) with all error components zero is considered. The estimated variance components are expressed as functions of the inter-item covariance matrix and the inter-examinee covariance matrix. These expressions show that under the random effects model if…

The one observation per cell two-way items by examinees random effects analysis of variance (ANOVA) with all error components zero is considered. The estimated variance components are expressed as functions of the inter-item covariance matrix and the inter-examinee covariance matrix. These expressions show that under the random effects model if…

The proposed coefficient is derived by assuming that the average Goodman-Kruskal gamma between items of identical difficulty would be the same for items of different difficulty. An estimate of covariance between items of identical difficulty leads to an estimate of the correlation between two tests with identical distributions of difficulty.…

This paper discusses optimal test construction, which deals with the selection of items from a pool to construct a test that performs optimally with respect to the objective of the test and simultaneously meets all test specifications. Optimal test construction problems can be formulated as mathematical decision models. Algorithms and heuristics…

This study was conducted to compare the usefulness of three item response theory (IRT) calibration packages (BILOG, BILOG-MG, and PIC) for examinations that include common and specialty components. Because small sample sizes and different mean abilities between specialty components are the most frequent problems that licensure/certification…

The research described in this paper shows the use of lexical semantic techniques for automated scoring of short-answer and essay responses from performance-based test items. Researchers used lexical semantic techniques in order to identify the meaningful content of free-text responses for small data sets. One data set involved 172 training…

Presented is an integer-programming approach to item pool design that can be used to calculate an optimal blueprint for an item pool to support an existing testing program. The results are optimal in the sense that they minimize the efforts involved in actually producing the items as revealed by current item writing patterns. Also presented is an…

This study evaluated the practical benefit, if any, of using collateral information for one item type when statistically analyzing pretest items of some other item type. The criterion for evaluation of pretest item calibration accuracy was the reduction achieved by the use of collateral information in the number of test takers that must be…

This study compared the ability of hierarchical linear modeling (HLM) to detect differential item functioning (DIF) to standard DIF detection methods, such as Rasch difficulty difference. The big advantages to using HLM for DIF detection are that the person abilities so produced are adjusted for any DIF in the items, and the DIF can then be…

The tendency for some survey respondents to be unreasonably agreeable on attitude measures with an agree-disagree format is typically termed acquiescence, or acquiescent responding (AR). This paper suggests an alternative operational definition of AR plus a statistical test for identifying AR subjects. If positively and negatively phrased items…

This paper addresses four steps in test construction specification: (1) the purpose of the test; (2) the content of the test; (3) the format of the test; and (4) the pool of items. If followed, such steps not only will assist the test constructor but will also enhance the students' learning. Within the "Content of the Test" section, two…

The term "equating" refers to a statistical procedure that adjusts test scores on different forms of the same examination so that scores can be interpreted interchangeably. This study examines the impact of equating with fewer items than originally planned when items have been removed from the equating set for a variety of reasons. A…