Conventional methods for evaluating the utility of subscores rely on traditional indices of reliability and on correlations among subscores. One limitation of correlational methods is that they do not explicitly consider variation in subtest means. An exception is an index of score profile reliability designated as [G], which quantifies the ratio…

This article extends multivariate generalizability theory (MGT) to tests with different random-effects designs for each level of a fixed facet. There are numerous situations in which the design of a test and the resulting data structure are not definable by a single design. One example is mixed-format tests that are composed of multiple-choice and…

Effects of different weighting schemes on selecting the optimal number of observations in multivariate-multifacet generalizability designs are studied when cost constraints are imposed. Comparison of four schemes through simulation indicates that all four produce similar optimal values and that reliability should be similar. (SLD)

The main purpose of this study is to examine the content structure of the Multistate Bar Examination (MBE) using the "table of specifications" model from the perspective of multivariate generalizability theory. Specifically, using MBE data collected over different years (six administrations: three from the February test and three from July test),…

A method is presented for determining the optimal number of conditions to use in multivariate-multifacet generalizability designs when resource constraints are imposed. A decision maker can determine the number of observations needed to obtain the largest possible generalizability coefficient. The procedure easily applies to the univariate case.…

Generalizability theory is used to examine the psychometric characteristics of the Listening and Writing Tests developed by American College Testing for its Work Keys program. Results with samples of 50 suggest the desirability of a minimum number of the tests' tape-recorded messages and the use of at least 2 raters. (SLD)