In this essay, we describe the construction and use of the Cut-Score Operating Function in aiding standard setting decisions. The Cut-Score Operating Function shows the relation between the cut-score chosen and the consequent error rate. It allows error rates to be defined by multiple loss functions and will show the behavior of each loss…

In this essay, the author tries to look forward into the 21st century to divine three things: (i) What skills will researchers in the future need to solve the most pressing problems? (ii) What are some of the most likely candidates to be those problems? and (iii) What are some current areas of research that seem mined out and should not distract…

This article presents an interview with Susan E. Embretson. Embretson attended the University of Minnesota where she received her bachelor's degree in 1967 and earned a PhD in 1973 in psychology. She became an assistant professor at the University of Kansas in 1974 and was promoted to associate professor and full professor. In 2004, she accepted a…

Confidence envelopes for one-parameter, two-parameter, and three-parameter logistic item response models are illustrated. M-line plots showing the genesis of the envelope and the density of lines in the confidence region are described and illustrated. (Author/SLD)

Proposes a parametric approach that involves a modification of standard Item Response Theory models that explicitly accounts for the nesting of items within the same testlets and that can be applied to multiple-choice sections comprising a mixture of independent items and testlets. (Author/SLD)

Describes an analytic method for aiding in the generation of subscores that characterize the deep structure of tests and derives a procedure for estimating scores for those scales that are more statistically stable than subscores composed solely of items contained on that scale. Used data from a Praxis administration (9,278 examinees) to show the…

This paper reviews the role of the item in test construction, and suggests some new methods of item analysis. A look at dynamic, graphical item analysis is provided that uses the advantages of modern, high-speed, highly interactive computing. Several illustrations are provided. (Author/TJH)

This study examined the Law School Admission Test (LSAT) through the use of testlet methods to model its inherent, locally dependent structure. Precision, measured by reliability, and fairness, measured by the comparability of performance across all identified subgroups of examinees, were the focus of the study. The polytomous item response theory…

A graphical-decision rule for studying an item's behavior and admitting items onto a test form is proposed. The tool--the isthmus of acceptance--ameliorates some parameter stability problems common to binary item response theory models by using a functional representation of item performance, and it is tuned to match current standards. (TJH)

The relationship between the multiple-choice and free-response sections of the Computer Science and Chemistry tests of the College Board's Advanced Placement program was studied. Confirmatory factor analysis showed that the free-response sections measure the same underlying proficiency as the multiple-choice sections for the most part. However,…

The testlet, a bundle of test items, alleviates some problems associated with computerized adaptive testing: context effects, lack of robustness, and item difficulty ordering. While testlets may be linear or hierarchical, the most useful ones are four-level hierarchical units, containing 15 items and partitioning examinees into 16 classes. (GDC)

A testlet is an integrated group of test items presented as a unit. The concept of testlet differential item functioning (testlet DIF) is defined, and a statistical method is presented to detect testlet DIF. Data from a testlet-based experimental version of the Scholastic Aptitude Test illustrate the methodology. (SLD)

It is sometimes sensible to think of the fundamental unit of test construction as being larger than an individual item. This unit, dubbed the testlet, must pass muster in the same way that items do. One criterion of a good item is the absence of differential item functioning (DIF). The item must function in the same way as all important…

Some cautions are sounded for converting a linearly administered test to an adaptive format. Four areas are identified in which practices broadly used in traditionally constructed tests can have adverse effects if thoughtlessly adopted when a test is administered in an adaptive mode. (SLD)

The National Assessment of Educational Progress (NAEP) is an enormous and enormously ambitious project. It generates data of a richness and complexity beyond any simple survey. The broad utilization of the information it provides can be aided through the use of more evocative data displays. In this report, the uses to which data tables are put is…