The use of multiple comparisons in analysis of variance (ANOVA) is discussed. It is argued that experimentwise Type I error rate inflation can be serious and that its influences are often unnoticed in ANOVA applications. Both classical balanced omnibus and orthogonal planned contrast tests inflate experimentwise error to an identifiable maximum.…

Various methods of estimating main effects from ordinal data are presented and contrasted. Problems discussed include: (1) at what level to accumulate ordinal data into linear measures; (2) how to maintain scaling across analyses; and (3) the inevitable confounding of within cell variance with measurement error. An example shows three methods of…

The rationale behind analysis of variance (including analysis of covariance and multiple analyses of variance and covariance) methods is reviewed, and unplanned and planned methods of evaluating differences between means are briefly described. Two advantages of using planned or a priori tests over unplanned or post hoc tests are presented. In…

Determination of statistical power for analysis of variance procedures requires five elements: (1) significance level; (2) effect size; (3) number of means; (4) error variance; and (5) sample size. Significance levels are traditionally chosen to be 0.5, .01, or .001. Effect size is not discussed in this paper. The number of means is determined by…

A method is presented for the simultaneous analysis of differential item functioning (DIF) in multi-factor situations. The method is unique in that it combines item response theory (IRT) and analysis of variance (ANOVA), takes a simultaneous approach to multifactor DIF analysis, and is capable of capturing interaction and controlling for possible…

The mixed model analysis of variance assumes a mathematical property known as sphericity. Several preliminary tests have been proposed to detect departures from the sphericity assumption. The logic of the preliminary testing procedure is to conduct the mixed model analysis of variance if the preliminary test suggests that the sphericity assumption…

In true experiments in which sample material can be randomly assigned to treatment conditions, most researchers presume that the condition of equal sample sizes is statistically desirable. When one or more a priori contrasts can be identified which represent a few overriding experimental concerns, however, allocating sample material unequally will…

Two bootstrapping or resampling strategies were investigated to determine their applicability to estimating standard errors and ensuing confidence intervals on variance components in two-factor random analysis of variance models. In light of prior negative findings regarding the application of bootstrapping to this particular problem, a…

The Type I error and power properties of the 2x2x2 analysis of variance (ANOVA) and tests developed by McSweeney (1967), Bradley (1979), Harwell-Serlin (1989; Harwell, 1991), and Blair-Sawilowsky (1990) were compared using Monte Carlo methods. The ANOVA was superior under the Gaussian and uniform distributions. The Blair-Sawilowsky test was…

The wide availability of large amounts of inexpensive computing power has encouraged statisticians to explore many approaches to a basis for inference. This paper presents one such "computer-intensive" approach: the bootstrap of Bradley Efron. This methodology fits between the cases where it is assumed that the form of the distribution…

This paper examined the rank transformation approach to analysis of variance as a solution to the Behrens-Fisher problem. Using simulation methodology four parameters were manipulated for the two group design: (1) ratio of population variances; (2) distribution form; (3) sample size and (4) population mean difference. The results indicated that…

Knowledge of the magnitude of effect(s) of an experimental study in science education should be of utmost concern to researchers in the field, but is often not reported. This document describes the concept of "explained variance" in analysis of variance designs and then explains how it can be calculated and reported. Reporting the magnitude of…

In educational research, nonessential factors are commonly ignored and when accounted for, they are often treated statistically as fixed effects. Yet many researchers in these situations generalize their findings beyond the specific levels selected; however, the analyses may require treating the factor as a random effect. Such inappropriate…

Most training programs in education and psychology focus on classical test theory techniques for assessing score dependability. This paper discusses generalizability theory and explores its concepts using a small heuristic data set. Generalizability theory subsumes and extends classical test score theory. It is able to estimate the magnitude of…

The nature and appropriate application of the technique of multivariate analysis are discussed. More specifically, the intent of the paper is to demystify and explain the use of multivariate analysis as well as provide guidelines for selection of the most effective statistics for use in specific situations. For the purpose of this paper, the term…