This comment discusses some issues about the use and reporting of experimental studies in education, illustrated by a recently published study that claimed (i) that an educational innovation was effective despite outcomes not reaching statistical significance, and (ii) that this refuted the findings of an earlier study. The two key issues raised…

In their response to our paper, Nicholson and Ridgway agree with the majority of what we wrote. They echo our concerns about the misuse of inferential statistics and NHST in particular. Very little of their response explicitly challenges the points we made but where it does their defence of the use of inferential techniques does not stand up to…

White and Gorard make important and relevant criticisms of some of the methods commonly used in social science research, but go further by criticising the logical basis for inferential statistical tests. This paper comments briefly on matters we broadly agree on with them and more fully on matters where we disagree. We agree that too little…

As an alternative to statistical testing, effect size has a non-monotonic linkage with practical importance. Besides random variance and systematic bias, a proper interpretation of effect size hinges on its implication to outcomes of deductive and/or inductive enquiries. Consequently, a small effect size might suggest an important finding, and the…

This comment offers three descriptions of "p[subscript rep]" that start with a frequentist account of confidence intervals, draw on R. A. Fisher's fiducial argument, and do not make Bayesian assumptions. Links are described among "p[subscript rep]," "p" values, and the probability a confidence interval will capture…

Considers objections to comparisonwise position, which holds that, when conducting simultaneous significance procedures, per-test Type I error rate should be controlled and that it is unnecessary to introduce adjustments designed to control familywise rate. Objections collected by Saville in an attempt to refute them are discussed along with…

D. Trafimow presented an analysis of null hypothesis significance testing (NHST) using Bayes's theorem. Among other points, he concluded that NHST is logically invalid, but that logically valid Bayesian analyses are often not possible. The latter conclusion reflects a fundamental misunderstanding of the nature of Bayesian inference. This view…

Responds to Biskin's comments (this issue) on the significance test controversy. Highlights areas of agreement (importance of replication evidence, importance of effect sizes) and disagreement (influence of sample size, evaluation of populations vs. samples, significance of Carver's article). Includes further recommendations for reporting research…

Significance tests are often used inappropriately in counseling research. In addition to comments on the significance test controversy (Vacha-Haase and Nilsson, this issue), a wider historical context and personal experiences are provided. The controversy is discussed, and seven recommendations for using significance tests are included. (EMK)

Discusses three topics related to interpretation of discriminant analyses (DA's): (1) partial F ratios and partial Wilks's lambdas for predictor variables in standard, step-down, and stepwise DA's; (2) relation of goals of classification to definition/evaluation of classification rules; and (3) significance tests for total hit rates in internal…

Makes the case that science is best served when researchers focus on the size of effects and their practical significance. Advocates the use of confidence intervals for deciding whether chance or sampling variability is an unlikely explanation for an observed effect. Calls for more emphasis on effect sizes in the next edition of the American…

A critical reanalysis of Barrett, Caldwell, and Alexander's (1985) critique of dynamic criteria. Summarizes and questions Barrett, et al.'s three definitions of dynamic criteria and their conclusion that reported temporal changes in criteria could be explained by methodological artifacts. A greater focus on dynamic criteria as constructs is…

Responds to Austin, Humphreys, and Hulin's (1989) critique of Barrett, Caldwell, and Alexander, suggesting that the burden of proof still rests on the advocates of the concept of dynamic criteria, and that empirical support is lacking for the existence of dynamic criteria as a simplex. Contrary evidence from educational, organizations, and…

This commentary on EC 603 695 argues that significance testing is a necessary but insufficient condition for positivistic research, that judgment-based assessment and single-subject research are not substitutes for significance testing, and that sampling fluctuation should be considered as one of numerous epistemological concerns in any…

Documents the recent use of one-tailed F-tests in communication journals, and examines the arguments both for and against their use. Examines the use of these tests within the broader unresolved controversy surrounding the use of one-tailed tests. Recommends that future researchers should most often avoid one-tailed Fs, and generally exercise…