This comment discusses some issues about the use and reporting of experimental studies in education, illustrated by a recently published study that claimed (i) that an educational innovation was effective despite outcomes not reaching statistical significance, and (ii) that this refuted the findings of an earlier study. The two key issues raised…

In their essay, "Why Assessment Will Never Work...," Bacon and Stewart (2016) recommend that instead of carrying out the expensive process of experimenting themselves, many business schools would get a bigger bang for their buck if they used "published pedagogical studies that use direct measures of learning with sufficient…

In their response to our paper, Nicholson and Ridgway agree with the majority of what we wrote. They echo our concerns about the misuse of inferential statistics and NHST in particular. Very little of their response explicitly challenges the points we made but where it does their defence of the use of inferential techniques does not stand up to…

This letter documents some problems in Ancaiani et al. (2015). Namely the evaluation of concordance, based on Cohen's kappa, reported by Ancaiani et al. was not computed on the whole random sample of 9,199 articles, but on a subset of 7,597 articles. The kappas relative to the whole random sample were in the range 0.07-0.15, indicating an…

White and Gorard make important and relevant criticisms of some of the methods commonly used in social science research, but go further by criticising the logical basis for inferential statistical tests. This paper comments briefly on matters we broadly agree on with them and more fully on matters where we disagree. We agree that too little…

In this interview, we discuss my early uses of meta-analytic procedures, first to combine "p"-values and then to combine effect sizes as well. My interest in quantifying the magnitude and the statistical significance of the effect of interpersonal expectations probably grew out of the following: (1) a long-held interest in the concept of…

In this report, a school choice advocacy group presents results from its survey of K-12 parents within and across the public and private sectors. They report that parents are highly satisfied with voucher and tax credit scholarship programs and suggest that the findings support the expansion of school choice programs. However, these and other…

Most biological scientists conduct experiments to look for effects, and test the results statistically. One of the commonly used test is Student's t test. However, this test concentrates on a very limited question. The authors assume that there is no effect in the experiment, and then estimate the possibility that they could have obtained these…

The authors of the study "The Long-Term Impact of Teachers" claim that their study shows that increases in teacher value-added lead to significant and lasting increases in test scores and significant increases in income that will last throughout adulthood. Instead, I show that these claims are false because they are contradicted by the…

The Coleman report, "Equality of Educational Opportunity," is the fountainhead for those committed to evidence-based education policy. Remarkably, this 737-page tome, prepared 50 years ago by seven authors under the leadership of James S. Coleman, still gets a steady 600 Google Scholar citations per year. But since its publication, views…

In his commentary, Foster (2010) made arguments at 2 levels, offering a broad critique of statistical or methodological approaches in developmental psychology in general together with critical comments that applied only to our recent article (Dogan, Stockdale, Widaman, & Conger, 2010). Certain criticisms by Foster aimed at the field as a whole…

A flood of contradictory publications claim widely different methods to be "evidence-based." What then are the key principles for success with challenging youth? Amidst all the confusion, how do we identify powerful universal truths? The answer lies in a "consilience," which involves drawing information from multiple fields of knowledge.…

As an alternative to statistical testing, effect size has a non-monotonic linkage with practical importance. Besides random variance and systematic bias, a proper interpretation of effect size hinges on its implication to outcomes of deductive and/or inductive enquiries. Consequently, a small effect size might suggest an important finding, and the…

This comment offers three descriptions of "p[subscript rep]" that start with a frequentist account of confidence intervals, draw on R. A. Fisher's fiducial argument, and do not make Bayesian assumptions. Links are described among "p[subscript rep]," "p" values, and the probability a confidence interval will capture…

In elementary books on applied statistics (e.g., Siegel, 1988; Agresti, 1990) and books on research methodology in psychology and personality assessment (e.g., Aiken, 1999), it is often suggested that the choice of a statistical test and the choice of statistical operations should be determined by the level of measurement of the data. Although…