The concept under investigation was the reliability of estimates of mean scores of groups under various assumptions of multiple-matrix sampling when reliabilities are computed according to procedures based on generalizability theory. Four different cases were compared with respect to the generalizability coefficients depending upon whether pupils…

Two-stage sampling procedures for comparing two population means when variances are heterogeneous have been developed by D. G. Chapman (1950) and B. K. Ghosh (1975). Both procedures assume sampling from populations that are normally distributed. The present study reports on the effect that sampling from non-normal distributions has on Type I error…

Multivariate methods are being used with increasing frequency in educational research because these methods control "experimentwise" error rate inflation, and because the methods best honor the nature of the reality to which the researcher wishes to generalize. This paper: explains the basic logic of canonical analysis; illustrates that…

Statistical significance is often misinterpreted to mean replicability or generalizability of results, although a statistically significant difference does not equal a reliable difference. Sample splitting procedures may be a more accurate way of estimating research result generalizability. This type of cross-validation involves randomly dividing…

A brief, practical overview of "design effects" (DEFFs) is presented for users of the results of sample surveys. The overview is intended to help such users to determine how and when to use DEFFs and to compute them correctly. DEFFs are needed only for inferential statistics, not for descriptive statistics. When the selections for…

This paper discusses the limitations of Classical Test Theory, the purpose of Item Response Theory/Latent Trait Measurement models, and the step-by-step calculations in the Rasch measurement model. The paper explains how Item Response Theory (IRT) transforms person abilities and item difficulties into the same metric for test-independent and…

An overview of the state of the art in psychological research is presented, with an emphasis on the attention given to effect sizes. The acceptance of small effect sizes for biomedical research is contrasted with the rejection of similar effect sizes for psychological research. The Binomial Effect Size Display is used to depict the practical…

Statistical procedures of the regression discontinuity and regression projection designs were compared to determine which design was more able to detect treatment effect in remedial or gifted programs. Neither test was more precise in all situations. Tables list the more precise test usder specific conditions. The regression projection model…

A study on sampling errors of variance components was conducted within the framework of generalizability theory by P. L. Smith (1978). The study used an intuitive approach for solving the problem of how to allocate the number of conditions to different facets in order to produce the most stable estimate of the universe score variance. Optimization…

A solution is proposed for analysis of variance procedures with missing cells, such as may occur when a control group is not assigned to any of the rows or columns of the various experimental groups. Mathematical models for two-way design are presented which define several variables; as well as row effect, column effect, and row and column…

A method is proposed for calculating the sample size needed to achieve acceptable statistical power with a given test. The minimally important difference significant (MIDS) criterion for sample size is explained and supported with recommendations for determining sample size. The MIDS criterion is computationally simple and easy to explain. (SLD)

In a situation where the population distribution of latent trait scores can be estimated, the ordinary maximum likelihood estimator of latent trait scores may be improved upon by taking the estimated population distribution into account. In this paper empirical Bayes estimators are compared with the liklihood estimator for three samples of 300…

A comparison was made between four different scoring procedures for the 45-item Reading Comprehension subtest, Level I, of the Comprehensive Test of Basic Skills, Form S, for a sample of 1,000 third grade students. These students were selected to have been among those who omitted from 3 to 22 of the 45 items. Another representative sample of 1,300…

The effects of using different data analysis methods on estimates of treatment effects of educational programs were investigated. Various regression models, such as those recommended for Title I program evaluations, were studied. The first effect studied was the amount of bias that might be expected to occur in the various settings. Results…

Because latent trait models require that large numbers of items be calibrated or that testing of the same large group be repeated, item parameter estimates are often obtained by administering separate tests to different groups and "linking" the results to construct an adequate item pool. Four issues were studied, based upon the analysis…