Composite scores are an exceptionally important psychometric tool for behavioral science research applications. A prototypical example occurs with self-report data, where researchers routinely use questionnaires with multiple items that tap into different features of a target construct. Item-level missing data are endemic to composite score…

This study investigated how the format of verbal instructions in computer simulations and prior knowledge (PK) affected 8th graders' cognitive load (CL) level and achievement in a multimedia learning environment. Although PK was not found to significantly affect student performance and CL level, instruction format was found to impact both.…

Drawing on game-design principles and an underlying situated theoretical perspective, we developed and researched a 3D game-based curriculum designed to teach water quality concepts. We compared undergraduate student dyads assigned randomly to four different instructional design conditions where the content had increasingly level of…

A computer simulation study was conducted to determine the feasibility of using logistic regression procedures to detect differential item functioning (DIF) in polytomous items. Results indicate that logistic regression is powerful in detecting most forms of DIF, although it requires large amounts of data manipulation and careful interpretation.…

Examined the degree to which coefficient alpha is affected by including items with different distribution shapes within a unidimensional scale. Computer simulation results indicate that reliability does not increase dramatically as a result of using differentially shaped items within a scale. Discusses implications for test construction. (SLD)

Exact testing was evaluated as a method for conducting Mantel-Haenszel differential item functioning (DIF) analyses with relatively small samples. A series of computer simulations found that the asymptotic Mantel-Haenszel and the exact method yielded very similar results across sample size, levels of DIF, and data sets. (SLD)

How item bias indexes based on item response theory (IRT) identify bias that results from multidimensionality is demonstrated. Simulation results suggest that IRT-based bias indexes detect multidimensional items with bias but do not detect multidimensional items without bias. They also do not confound between-group differences on the primary test.…

This article proposes an item exposure control method, which is the extension of the Sympson and Hetter procedure and can provide item exposure control at both the item and test levels. Item exposure rate and test overlap rate are two indices commonly used to track item exposure in computerized adaptive tests. By considering both indices, item…

Sequential probability ratio testing (PRT), which usually is applied in situations requiring a decision between two simple hypotheses or a single decision point, is extended to include situations involving k decision points and [(k + 1)-choose-2] sets of simultaneous, simple hypotheses, where k>1. The multiple-decision point or…

The equating of results from the PC-BILOG computer program to an underlying metric was studied through simulation when a two-parameter item response theory model was used. Results are discussed in terms of the identification problem and implications for test equating. (SLD)

It is demonstrated, theoretically and empirically, that item sets can be selected that meet the unidimensionality assumption of most item response theory models, even though they require more than one ability for a correct response. A method for identifying such item sets for test development purposes is presented. (SLD)

The Hetter and Sympson (1997; 1985) method is a method of probabilistic item-exposure control in computerized adaptive testing. Setting its control parameters to admissible values requires an iterative process of computer simulations that has been found to be time consuming, particularly if the parameters have to be set conditional on a realistic…

This study used innovative assessment practices to obtain and document broad learning outcomes for a 15-hour game-based curriculum in Quest Atlantis, a multi-user virtual environment that supports school-based participation in socio scientific inquiry in ecological sciences. Design-based methods were used to refine and align the enactment of…

The capability of the DIMTEST statistical test to assess essential dimensionality of the model underlying item responses of real tests as opposed to simulated tests was investigated. A variety of real test data from difference sources was used to assess essential dimensionality. Based on DIMTEST results, some test data are assessed as fitting an…

This simulation investigated two procedures that reduce differences between paper-and-pencil testing and computerized adaptive testing (CAT) by making CAT content sensitive. Results indicate that the price in terms of additional test items of using constrained CAT for content balancing is much smaller than that of using testlets. (SLD)