Examined the stability of the standard error of measurement and the relationship between the reliability coefficient and the variance of both true scores and error scores for 170 school districts in a state. As expected, reliability coefficients varied as a function of group variability, but the variation in split-half coefficients from school to…

Examined whether the item response theory (IRT) true-score equating method is more adversely affected by the presence of multidimensionality than two conventional equating methods, linear and equipercentile equating. Results of two simulation studies suggest that the IRT method performs as well as the conventional methods when the correlation…

Four equating methods (3PL true score equating, 3PL observed score equating, beta 4 true score equating, and beta 4 observed score equating) were compared using four equating criteria: first-order equity (FOE), second-order equity (SOE), conditional-mean-squared-error (CMSE) difference, and the equi-percentile equating property. True score…

A decision-theoretic approach to the question of reliability in categorically scored examinations is explored. The concepts of true scores and errors are discussed as they deviate from conventional psychometric definitions and measurement error in categorical scores is cast in terms of misclassifications. A reliability measure based on…

This article describes procedures for estimating various indices of classification consistency and accuracy for multiple category classifications using data from a single test administration. The estimates of the classification consistency and accuracy indices are compared under three different psychometric models: the two-parameter beta binomial,…

This article studies the ordinal reliability of (total) test scores. This study is based on a classical-type linear model of observed score (X), true score (T), and random error (E). Based on the idea of Kendall's tau-a coefficient, a measure of ordinal reliability for small-examinee populations is developed. This measure is extended to large…

This study evaluated the impact of unequal reliability on test equating methods in the nonequivalent groups with anchor test (NEAT) design. Classical true score-based models were compared in terms of their assumptions about how reliability impacts test scores. These models were related to treatment of population ability differences by different…

A new formula is developed for the relative efficiency of two tests measuring the same trait. The formula expresses relative efficiency solely in terms of the standard errors of measurement and, surprisingly, the frequency distributions of true scores. Approximate methods for estimating relative efficiency may make this function routinely…

The relationship between true ability (operationally defined as the number of items for which the examinee actually knew the correct answer) and the effects of guessing upon observed test variance was investigated. Three basic hypotheses were treated mathematically: there is no functional relationship between true ability and guessing success;…

In a post mortem study, it is demonstrated that linear prediction is as effective as computing a negative hyper-geometric distribution for estimating test norms following matrix sampling from a total test with a highly skewed score distribution, provided the same prediction coefficient is used for all examinee groups. It is also demonstrated…

A general coefficient for tests, delta, is derived from a decision theoretic point of view. The situations are considered in which a true score is estimated by a function of the observed score, observed scores are split into more than two categories, and observed scores are split into only two categories. (Author/CTM)

A computer simulation study was conducted to investigate Subkoviak's index of reliability for criterion-referenced tests, called the coefficient of agreement. Results indicate that the index can be adequately estimated. (JKS)

A test of significance for identifying individuals who are most influenced by an experimental treatment as measured by pre-post test change score is presented. The technique requires true difference scores, the reliability of obtained differences, and their standard error of measurement. (Author/JKS)

Derives formulas for the validity of predictor-criterion tests that hold for all test scores constructed according to the expected-value concept of true score. These more general formulas disclose some paradoxical properties of test validity under conditions where errors are correlated and have some implications for practical testing situations…

This article is concerned with estimation of components of maximum generalizability in multifacet experimental designs involving multiple dependent measures. An example of a two-facet partially nested design is provided. (Author/RC)