Reflections on the efficacy of pedagogical changes and practices and their effect on student performance are often hindered by incomplete data, small sample sizes, and the confounding variables of multiple instructors and teaching sites. Observations from such retrospective analyses, however, are highly sought after by instructors and…

The recent simulation-based inference (SBI) movement in algebra-based introductory statistics courses (Stat 101) has provided preliminary evidence of improved student conceptual understanding and retention. However, little is known about whether these positive effects are preferentially distributed across types of students entering the course. We…

Changes in medical school curricula often require educators to develop teaching strategies that decrease contact hours while maintaining effective pedagogical methods. When faced with this challenge, faculty at the University of Cincinnati College of Medicine converted the majority of in-person histology laboratory sessions to self-study modules…

Problem Statement: Computerized adaptive testing (CAT) is a sophisticated and efficient way of delivering examinations. In CAT, items for each examinee are selected from an item bank based on the examinee's responses to the items. In this way, the difficulty level of the test is adjusted based on the examinee's ability level. Instead of…

This article presents a new algorithm for computerized adaptive testing (CAT) when content constraints are present. The algorithm is based on shadow CAT methodology to meet content constraints but applies Monte Carlo methods and provides the following advantages over shadow CAT: (a) lower maximum item exposure rates, (b) higher utilization of the…

Kernel smoothing methods for nonparametric item characteristic curve estimation are reviewed. A simulation with 500 examinees and real data from 3,000 records of the Graduate Record Examination illustrate the rapidity of kernel smoothing. Even when population curves are three-parameter logistic, simulation suggests no loss of efficiency. (SLD)

A procedure--based on item response theory--for testing the hypothesis of unidimensionality of the latent space is proposed. Use of the procedure is supported by an asymptotic theory and a Monte Carlo simulation study. The procedure tests for unidimensionality in test construction and/or compares two tests. (SLD)

In item response theory (IRT), preequating depends upon item parameter estimate invariance. Three separate simulations, all using the unidimensional three-parameter logistic item response model, were conducted to study the impact of the following variables on preequating: (1) mean differences in ability; (2) multidimensionality in the data; and…

Six procedures for combining sets of item response theory (IRT) item parameter estimates from different samples were evaluated using real and simulated response data. Results support use of covariance matrix-weighted averaging and a procedure using sample-size-weighted averaging of estimated item characteristic curves at the center of the ability…

This paper reviews the basic elements of the EM approach to estimating item parameters and illustrates its use with one simulated and one real data set. In order to illustrate the use of the BILOG computer program, runs for 1-, 2-, and 3-parameter models are presented for the two sets of data. First is a set of responses from 1,000 persons to five…

The extensive computer simulation work done in developing the computer adaptive versions of the Graduate Record Examinations (GRE) Board General Test and the College Board Admissions Testing Program (ATP) Scholastic Aptitude Test (SAT) is described in this report. Both the GRE General and SAT computer adaptive tests (CATs), which are fixed length…

The consequences in terms of misclassifications of students that would occur by replacing the constructed-response portion of the Advanced Placement (AP) examinations in economics with more multiple-choice items were studied. The 1991 AP examinations in micro- and macroeconomics were used. Computer simulation found that a small but statistically…

The purpose of this study was to investigate the effect of using multidimensional items in a computer adaptive test (CAT) setting which assumes a unidimensional item response theory (IRT) framework. Previous research has suggested that the composite of multidimensional abilities being estimated by a unidimensional IRT model is not constant…

A popular method of analyzing test items for differential item functioning (DIF) is to compute a statistic that conditions samples of examinees from different populations on an estimate of ability. This conditioning or matching by ability is intended to produce an appropriate statistic that is sensitive to true differences in item functioning,…

Some examinees' test-taking behavior may be so idiosyncratic that their scores are not comparable to the scores of more typical examinees. Appropriateness indices, which provide quantitative measures of response-pattern atypicality, can be viewed as statistics for testing a null hypothesis of normal test-taking behavior against an alternative…