Today's health sciences educational programmes have to deal with a growing and changing amount of knowledge. It is becoming increasingly important for students to be able to use and manage knowledge. We suggest incorporating open-book tests in assessment programmes to meet these changes. This view on the use of open-book tests is discussed and the…

Multivariate methods are being used with increasing frequency in educational research because these methods control "experimentwise" error rate inflation, and because the methods best honor the nature of the reality to which the researcher wishes to generalize. This paper: explains the basic logic of canonical analysis; illustrates that…

Generalizability or invariance procedures have been known for over three decades. Through the years, these procedures have not been widely discussed or employed. One reason for the lack of use is that most of the articles on invariance procedures have been mathematically oriented. The mathematical orientation of research articles and the lack of…

Generalizability or invariance procedures have been known for over three decades. Through the years, these procedures have not been widely discussed or employed. One reason for the lack of use is that most of the articles on invariance procedures have been mathematically oriented. The mathematical orientation of research articles and the lack of…

The present paper suggests that multivariate techniques are very important in social science research, and that canonical correlation analysis may be particularly useful. The logic of canonical analysis is explained and discussed. The necessity of using replicability/generalizability analyses is argued. It is suggested that cross-validation…

The main purpose of this study is to examine the content structure of the Multistate Bar Examination (MBE) using the "table of specifications" model from the perspective of multivariate generalizability theory. Specifically, using MBE data collected over different years (six administrations: three from the February test and three from July test),…

Statistical invariance procedures provide a way of looking at the generalizability of research results from sample to sample when the research has not been validated by replication. This paper discusses the Procrustean Rotation invariance procedure following a canonical correlation analysis. The computer program RELATE is used to gauge the…

The need for using invariance procedures to establish the external validity or generalizability of statistical results has been well documented. Invariance analysis is a tool that can be used to establish confidence in the replicability of research findings. Several approaches to invariance analysis are available that are broadly applicable across…

A method is presented for determining the optimal number of conditions to use in multivariate-multifacet generalizability designs when resource constraints are imposed. A decision maker can determine the number of observations needed to obtain the largest possible generalizability coefficient. The procedure easily applies to the univariate case.…

In this study, the reliability of the American College Testing (ACT) Program's "Survey of Academic Advising" (SAA) was examined using both univariate and multivariate generalizability theory approaches. The primary purpose of the study was to compare the results of three generalizability theory models (a random univariate model, a mixed…

Tables of specifications are used to guide test developers in sampling items and maintaining consistency from form to form. This paper is a generalizability study of the American College Testing Program (ACT) Achievement Program Mathematics Test (AAP), with the content areas of the table of specifications representing multiple dependent variables.…

Generalizability theory is used to examine the psychometric characteristics of the Listening and Writing Tests developed by American College Testing for its Work Keys program. Results with samples of 50 suggest the desirability of a minimum number of the tests' tape-recorded messages and the use of at least 2 raters. (SLD)