This paper follows a line of logical argumentation to claim that what Samuel Messick conceptualized about construct validation has probably been misunderstood by some educational policy makers, practicing educators, and classroom teachers. It argues that, while Messick's unified theory of test validation aimed at (a) warning educational…

In adolescence research, the treatment of measurement reliability is often fragmented, and it is not always clear how different reliability coefficients are related. We show that generalizability theory (G-theory) is a comprehensive framework of measurement reliability, encompassing all other reliability methods (e.g., Pearson "r,"…

The Texas State University-San Marcos undergraduate healthcare administration program requires all bachelors of health administration (BHA) students to pass a comprehensive examination to demonstrate their knowledge of specific core competencies. This also demonstrates completion of their didactic coursework in order to enter a practical…

In educational testing, subscores may be provided based on a portion of the items from a larger test. One consideration in evaluation of such subscores is their ability to predict a criterion score. Two limitations on prediction exist. The first, which is well known, is that the coefficient of determination for linear prediction of the criterion…

Classical procedures for calculating the two indices of decision consistency (P and Kappa) for criterion-referenced tests require two testings on each child. Huynh, Peng, and Subkoviak have presented one-testing procedures for these indices. These indices can be estimated without any test administration using Ebel's estimates of the mean, standard…

Four criterion-referenced reliability coefficicents were compared to the Kuder-Richardson estimates and to each other. The Kuder-Richardson formulas 20 and 21, the Livingston, the Subkoviak and two Huynh coefficients were computed for a random sample of 33 criterion-referenced tests. The Subkoviak coefficient yielded the highest mean value;…

Binomial models for domain-referenced testing are compared, emphasizing the assumptions underlying the beta-binomial model. Advantages and disadvantages are discussed. A proposed item sampling model is presented which takes the effect of guessing into account. (Author/CM)

The paper considers the problem of selecting the t best of k normal populations and simultaneously determining whether the selected populations have a mean larger than a known standard. Illustrations are given for selecting the t best of k examinees when the binomial error model applies. (Author)

Focusing on the use of "paper and pencil" criterion-referenced tests in educational measurement, and to correct misconceptions, the definitions of basic terms and historical antecedents are discussed. Classifications of the tests are compared with other achievement tests. The phases in developing criterion-referenced tests are presented with the…

The relationship between classical discrimination indices (CDI) and criterion-referenced discrimination indices (CRDI) and the appropriateness of each for use on criterion-referenced tests are investigated. A CRDI is proposed that attempts to separate those who master material from those who do not master material. A 26 item multiple-choice…

Basic issues in criterion-referenced measurement are addressed. In section II, issues involved in determining what a person does and can do are considered. A preliminary analysis of "can" is given which shows that there are several important senses of "can". In section III, results of an analysis of "ability" are…

Three item selection methods for criterion-referenced tests are examined: the classical theory of item difficulty and item-test correlation; the latent trait theory of item characteristic curves; and a decision-theoretic approach for optimal item selection. Item contribution to the standardized expected utility of mastery testing is discussed. (CM)

Mastery learning strategies and criterion referenced measurement tools perform a selective function in the classroom. The selective approach within the philosophical role of schools is discussed in terms of limited educational employment, competition, talent distribution, and the suggested attributes of good testing. (CM)

For some situations the beta-binomial distribution might be used to describe the marginal distribution of test scores for a particular population of examinees. Several different methods of approximating the maximum likelihood estimate were investigated, and it was found that the Newton-Raphson method should be used when it yields admissable…

This paper evaluates the magnitude of the total error in estimates of the difference between an examinee's domain score and the cutoff score. An observed score based on a random sample of items from the domain, and an estimated cutoff score derived from a judgmental standard setting procedure are assumed. The work of Brennan and Lockwood (1980) is…