Livingston's reliability coefficients and Harris' indices of efficiency were computed along with the classical internal consistency coefficients, KR-20's (Kuder-Richardson internal consistency coefficient), for 678 criterion-referenced tests in the A through E levels of an individualized mathematics program. The coefficients were carefully studied…

Besides the ubiquitous Pearson product-moment r, there are a number of other measures of relationship that are attenuated by errors of measurement and for which the relationship between true measures can be estimated. Among these are the correlation ratio (eta squared), Kelley's unbiased correlation ratio (epsilon squared), Hays' omega squared,…

Since the primary purpose of classical testing is to rank order examinees consistently, the absolute value of the true score has been relatively unimportant. However, the major purpose of criterion referenced testing is to estimate the true capabilities of examinees to perform specific tasks. Hence, the problems of true score determination assume…

The Law School Admission Test (LSAT) was examined to see if the items on a form could be divided into different subgroups in which items looked statistically similar within the subgroups but statistically different between subgroups. Of such subgrouping can be detected, it is likely that the subgroups of items measure different abilities, and the…

Test-retest correlations can lead to biased reliability estimates when there is instability of true scores and/or when measurement errors are correlated. Using three administrations of the Test of Standard Written English and essay ratings, an analysis is demonstrated which separates true score instability and correlated errors. (Author/BW)

For anchor test equating, 3 linear observed score methods are derived for populations differing in ability. Each version requires that the correlations of the tests with the selection variable be known. Five sets of assumptions are made for each model--yielding 15 methods--which are then related to existing methods. (SLD)

The current research was conducted to investigate the validity of automated essay scoring (AES) by comparing group mean scores assigned by an AES tool, IntelliMetric [TM] and human raters. Data collection included administering the Texas version of the WriterPlacer "Plus" test and obtaining scores assigned by IntelliMetric [TM] and by…

Much previously published material for estimating the reliability of classification has been based on the assumption that a test consists of a known number of equally weighted items. The test score is the number of those items answered correctly. These methods cannot be used with classifications based on weighted composite scores, especially if…

Because it is a true score model employing item parameters which are independent of the examined sample, item characteristic curve theory (ICC) offers several advantages over classical measurement theory. In this paper an approach to biased item identification using ICC theory is described and applied. The ICC theory approach is attractive in that…

A probabilistic hypothesis testing procedure to assess the fit of hypothesized hierarchical structures for test item data is discussed. Statistical procedures are presented which are useful for evaluating the fit of data of a certain class of probabilistic models. These models apply to sets of dichotomous (O,1) responses for which there are…

The Bayesian approach to setting passing scores, as proposed by Swaminathan, Hambleton, and Algina, is compared with the empirical Bayes approach to the same problem that is derived from Huynh's decision-theoretic framework. Comparisons are based on simulated data which follow an approximate beta-binomial distribution and on real test results from…

Livingston's work is a careful analysis of what occurs when one pools two populations with different means, but similar variances and reliability coefficients. However, his work fails to advance reliability theory for the special case of criterion-referenced testing. See ED 042 802 for Livingston's paper. (MS)

Intraclass correlation reliability estimates are based on the assumption that the various measures are equivalent. Joreskog's (1970) general model for the analysis of covariance structures can be used to test the validity of this assumption. (For related document, see TM 002 301.) (Author)