For efficiency, Computerized Adaptive Test (CAT) algorithm selects items with the maximum information, typically with a 50% probability of being answered correctly. However, examinees may not be satisfied if they only correctly answer 50% of the items. Researchers discovered that changing the item selection algorithms to choose easier items (i.e.,…

The research shared in this conference paper report illustrates how an iterative process to item development that involves expert review and cognitive lab interviews with students can be used to collect evidence of validity for assessment items. Analysis of students' reasoning was also used to expand a model for identifying conceptions and…

This paper attempts to quantify the accuracy limit of "nextitem-correct" prediction by using numerical optimization to estimate the student's probability of getting each question correct given a complete sequence of item responses. This optimization is performed without an explicit parameterized model of student behavior, but with the…

Basic issues in understanding Item Response Theory (IRT), or Latent Trait Theory, measurement models are discussed. These theories have gained popularity because of their promise to provide greater precision and control in measurement involving both achievement and attitude instruments. IRT models implement probabilistic techniques that yield…

Educational assessment is a process of collecting evidence and interpreting it to provide instructors with information regarding students' learning. However, the current design and scoring of most standardized educational tests are insufficient to serve this purpose. The limitation exists primarily due to the lack of cognitive information…

One challenge associated with computerized adaptive testing (CAT) is the maintenance of test and item security while allowing for daily testing. An alternative to continually creating new pools containing an independent set of items would be to consider each CAT pool as a sample of items from a larger collection (referred to as a VAT) rather than…

This study investigated the impact of task directions on the mathematical performance of high school students from six classes. Students analyzed data regarding school dropout by answering six short-answer questions and writing a letter discussing the trends and their predictions about school dropout. Tasks were scored using two methods: (1) trait…

Three methods of factor analyzing dichotomously scored item performance data were compared using two raw score data sets of 20-item tests, one reflecting normally distributed latent traits and the other reflecting uniformly distributed latent traits. This comparison was accomplished by using phi and tetrachoric correlations among dichotomous data…

Two experiments reproduced in a simulated computerized test-taking situation the effect of two of the main determinants in answering an item in a test: the difficulty of the item and the time available to answer it. A model is proposed for the time to respond or abandon an item and for the probability of abandoning it or answering it correctly. In…

To generalize test results beyond the particular test administration, an examinee's ability estimate must be independent of the particular items attempted, and the item difficulty calibrations must be independent of the particular sample of people attempting the items. This stability is a key concept of the Rasch model, a latent trait model of…

DIMTEST is a statistical test procedure for assessing essential unidimensionality of binary test item responses. The test statistic T used for testing the null hypothesis of essential unidimensionality is a nonparametric statistic. That is, there is no particular parametric distribution assumed for the underlying ability distribution or for the…

The analysis of fit of data to a measurement model for graded responses is described. The model is an extension of Rasch's dichotomous model to formats which provide more than two levels of response to items. The model contains one parameter for each person and one parameter for each "step" in an item. A dichotomously-scored item…

A latent trait method is presented to investigate the possibility that Angoff or Nedelsky judges specify inconsistent probabilities in standard setting techniques for objectives-based instructional programs. It is suggested that judges frequently specify a low probability of success for an easy item but a large probability for a hard item. The…

Effects of altering test difficulty on examinee ability measures and test length in a computer adaptive test were studied for 225 medical technology students in 3 test difficulty conditions. Results suggest that, with an item pool of sufficient depth and breadth, acceptable targeting to test difficulty is possible. (SLD)

A scaling procedure is proposed, based on item response theory (IRT), to fit non-hierarchical test structure as well. The binary scores of a test of English were used for calculating the probabilities of answering each item correctly. The probability matrix was factor analyzed, and the difficulty intervals or estimates corresponding to the factors…