This report describes the derivation and evaluation of a method for comparing the performance standards for public school students set by different states. It is based on an approach proposed by McLaughlin and associates, which constituted an innovative attempt to resolve the confusion and concern that occurs when very different proportions of…

This technical report describes the development of reading comprehension assessments designed for use as progress monitoring measures appropriate for 2nd Grade students. The creation, piloting, and technical adequacy of the measures are presented. The following are appended: (1) Item Specifications for MC [Multiple Choice] Comprehension - Passage…

This technical report describes the development and piloting of reading comprehension measures developed for use by fifth-grade students as part of an online progress monitoring assessment system, http://easycbm.com. Each comprehension measure is comprised of an original work of narrative fiction approximately 1500 words in length followed by 20…

The use of microworlds (MWs), or complex dynamic systems, in educational testing and personnel selection is hampered by systematic measurement errors because these new and innovative item formats are not adequately controlled for their difficulty. This empirical study introduces a way to operationalize an MW's difficulty and demonstrates the…

This study examines how closely the kernel equating (KE) method (von Davier, Holland, & Thayer, 2004a) approximates the results of other observed-score equating methods--equipercentile and linear equatings. The study used pseudotests constructed of item responses from a real test to simulate three equating designs: an equivalent groups (EG)…

Two IRT-based procedures to estimate test reliability for a certification exam that used both adaptive (via a MST model) and non-adaptive design were considered in this study. Both procedures rely on calibrated item parameters to estimate error variance. In terms of score variance, one procedure (Method 1) uses the empirical ability distribution…

In this simulation study, we investigate the power and Type I error rate of a procedure based on the mean and covariance structure analysis (MACS) model in detecting differential item functioning (DIF) of graded response items with five response categories. The following factors were manipulated: type of DIF (uniform and non-uniform), DIF…

Structural equation modeling allows several methods of estimating the disattenuated association between 2 or more latent variables (i.e., the measurement model). In one common approach, measurement models are specified using item parcels as indicators of latent constructs. Item parcels versus original items are often used as indicators in these…

The computation and reporting of effect size estimates is becoming the norm in many journals in psychology and related disciplines. Despite the increased importance of effect sizes, researchers may not report them or may report inaccurate values because of a lack of appropriate computational tools. For instance, Pierce, Block, and Aguinis (2004)…

Cross-classified random effects modeling (CCREM) is used to model multilevel data from nonhierarchical contexts. These models are widely discussed but infrequently used in social science research. Because little research exists assessing when it is necessary to use CCREM, 2 studies were conducted. A real data set with a cross-classified structure…

Various methods of estimating main effects from ordinal data are presented and contrasted. Problems discussed include: (1) at what level to accumulate ordinal data into linear measures; (2) how to maintain scaling across analyses; and (3) the inevitable confounding of within cell variance with measurement error. An example shows three methods of…

Tukey's Honestly Significant Difference (HSD) procedure (J. Tukey, 1953) is probably the most recommended and used procedure for controlling Type I error rate when making multiple pairwise comparisons as follow-ups to a significant omnibus F test. This study compared observed Type I errors with nominal alphas of 0.01, 0.05, and 0.10 compared for…

This repeated measures factorial design study compared the results of two procedures for identifying multivariate outliers under varying conditions, the Mahalanobis distance and the Andrews-Pregibon statistic. Results were analyzed for the total number of outliers identified and number of false outliers identified. Simulated data were limited to…

A brief, practical overview of "design effects" (DEFFs) is presented for users of the results of sample surveys. The overview is intended to help such users to determine how and when to use DEFFs and to compute them correctly. DEFFs are needed only for inferential statistics, not for descriptive statistics. When the selections for…

This study applies a procedure which yields estimates of true score change on the Scholastic Aptitude Test (SAT) adjusted for regression effects and student self-selection. It is shown that student self-selection in deciding to repeat an admissions test probably involves factors in addition to the measurement error attributable to variations in…