Students often believe that statistical significance is the only determinant of whether a quantitative result is "important." In this paper, I review traditional null hypothesis statistical testing to identify what questions inferential statistics can and cannot answer, including statistical significance, effect size and direction,…

In their response to our paper, Nicholson and Ridgway agree with the majority of what we wrote. They echo our concerns about the misuse of inferential statistics and NHST in particular. Very little of their response explicitly challenges the points we made but where it does their defence of the use of inferential techniques does not stand up to…

A multiple testing method for examining factorial invariance for latent constructs evaluated by multiple indicators in distinct populations is outlined. The procedure is based on the false discovery rate concept and multiple individual restriction tests and resolves general limitations of a popular factorial invariance testing approach. The…

This brief paper introduces a new approach to assessing the trustworthiness of research comparisons when expressed numerically. The 'number needed to disturb' a research finding would be the number of counterfactual values that can be added to the smallest arm of any comparison before the difference or 'effect' size disappears, minus the number of…

A classic TV commercial once asked, "How many licks does it take to get to the center of a Tootsie Roll Tootsie Pop?" The narrator claims, "The world may never know" (Tootsie Roll 2012), but an Internet search returns a multitude of answers, some of which include rigorous systematic approaches by academics to address the…

The self-correcting nature of psychological and educational science has been seriously questioned. Recent special issues of "Perspectives on Psychological Science" and "Psychology of Aesthetics, Creativity, and the Arts" have roundly condemned current organizational models of research and dissemination and have criticized the…

This report introduces the "WorkFORCE"™ Assessment for Job Fit, a personality assessment utilizing the "FACETS"™ core capability, which is based on innovations in forced-choice assessment and computer adaptive testing. The instrument is derived from the fivefactor model (FFM) of personality and encompasses a broad spectrum of…

Asterisks should not be used to indicate if the result of a hypothesis test is significant.

A quiet methodological revolution, a modeling revolution, has occurred over the past several decades, almost without discussion. In contrast, the 20th century ended with contentious argument over the utility of null hypothesis significance testing (NHST). The NHST controversy may have been at least partially irrelevant, because in certain ways the…

Null distributions of permutation tests for two-sample, paired, and block designs are simulated using the R statistical programming language. For each design and type of data, permutation tests are compared with standard normal-theory and nonparametric tests. These examples (often using real data) provide for classroom discussion use of metrics…

We discuss the identification, estimation, and testing of structural equation models that have causal indicators. We first provide 2 rules of identification that are particularly helpful in models with causal indicators--the 2C emitted paths rule and the exogenous X rule. We demonstrate how these rules can help us distinguish identified from…

This paper proposes an argument framework for the teaching of null hypothesis statistical testing and its application in support of research. Elements of the Toulmin (1958) model of argument are used to illustrate the use of p values and Type I and Type II error rates in support of claims about statistical parameters and subject matter research…

This article discusses issues associated with statistical testing conducted with data from clustered school samples. Empirical researchers often conduct tests of statistical inference on sample data to ascertain the extent to which differences exist within groups in the population. Typically, much school-related data are collected from students.…

The National Assessment of Educational Progress (NAEP) is a continuing and nationally representative assessment of what this nation's students know and can do. NAEP has often been called the "gold standard" of assessments because it is developed using the best thinking of assessment and content specialists, education experts, and…

Very little explanatory power is required in order for regressions to exhibit statistical significance. This article discusses some of the causes and implications. (Contains 2 tables.)