Previous investigations of the ability of content experts and test developers to estimate item difficulty have, for themost part, produced disappointing results. These investigations were based on a noncomparative method of independently rating the difficulty of items. In this article, we argue that, by eliciting comparative judgments of…

Higher education is deploying a range of innovative strategies in response to rising costs and financial challenges. This report presents findings from a NASPA [National Association of Student Personnel Administrators] study on cost-saving initiatives at three institutions. Each institution aimed to creatively address different areas affecting…

Recent simulation research has demonstrated that using simple raw score to operationalize a latent construct can result in inflated Type I error rates for the interaction term of a moderated statistical model when the interaction (or lack thereof) is proposed at the latent variable level. Rescaling the scores using an appropriate item response…

A number of vertical scaling methodologies were examined in this article. Scaling variations included data collection design, scaling method, item response theory (IRT) scoring procedure, and proficiency estimation method. Vertical scales were developed for Grade 3 through Grade 8 for 4 content areas and 9 simulated datasets. A total of 11 scaling…

This study compared the performance of the LOGIST and BILOG computer programs on item response theory (IRT) based scaling and equating for the Test of English as a Foreign Language (TOEFL) using real and simulated data and two calibration structures. Applications of IRT for the TOEFL program are based on the three-parameter logistic (3PL) model.…

Using simulated data modeled after real tests, a Thurstone method (L. Thurstone, 1925 and later) and three-parameter item response theory were compared for vertical scaling. Neither procedure produced artificial scale shrinkage, and both produced modest scale expansion for one simulated condition. (SLD)

The effect of scale selection on error in calibrating item and ability parameters was investigated, with particular reference to the standardized mean-squared difference (SMSD) statistic. Through simulation, three scaling methods for selecting the common scale were used to demonstrate their effects on SMSD values. (SLD)

Examined the degree to which coefficient alpha is affected by including items with different distribution shapes within a unidimensional scale. Computer simulation results indicate that reliability does not increase dramatically as a result of using differentially shaped items within a scale. Discusses implications for test construction. (SLD)

This study compared vertical scaling results for the Rasch model from BILOG-MG and WINSTEPS. The item and ability parameters for the simulated vocabulary tests were scaled across 11 grades; kindergarten through 10th. Data were based on real data and were simulated under normal and skewed distribution assumptions. WINSTEPS and BILOG-MG were each…

A method for robust canonical discriminant analysis via two robust objective loss functions is discussed. Majorization is used at several stages in the minimization procedure to obtain a monotonically convergent algorithm. A simulation study and empirical data illustrate the procedure. (SLD)

To solve an applied research problem with bipolar data, a set of equations was developed that combined all plus and minus data combinations into unique values and scale points. The equations were tested through computer simulations and empirical tests. Resulting index scores were approximately interval and linear and easy to use and interpret.…

How common patterns of rater errors may be detected in a large-scale performance assessment setting is discussed. Common rater effects are identified, and a scaling method that can be used to detect them in operational data sets is presented. Simulated data sets are generated to exhibit each of these rater effects. The three continua that depict…

One-parameter (1PPC) and two-parameter partial credit (2PPC) models were compared using real and simulated data with constructed response items present. Results suggest that the more flexible three-parameter logistic-2PPC model combination produces better model fit than the combination of the one-parameter logistic and the 1PPC models. (SLD)

A technique is presented to transform incomplete categorical data into complete data by imputing appropriate scores into missing cells. A solution of the optimization problem is suggested, and relevant psychometric theory is discussed. The average correlation should be at least 0.50 before the method becomes practical. (SLD)

The use of two alternative item response theory (IRT) estimation models in the scaling and equating of the Test of English as a Foreign Language (TOEFL) was explored; and item scaling and test equating results based on these models were compared with results based on the three-parameter (3PL) model currently being used with the TOEFL. Models were…