The import or force of the result of a statistical test has long been portrayed as consistent with deductive reasoning. The simplest form of deductive argument has a first premise with conditional form, such as p[right arrow]q, which means that "if p is true, then q must be true." Given the first premise, one can either affirm or deny…

Common sense is a dynamic concept and it is natural that our (statistical) common sense lags behind the development of statistical science. What is not so easy to understand is why common sense lags behind as much as it does. We conduct a survey among Japanese students and provide examples and tentative explanations of a number of statistical…

Considered normative from the second half of the 20th century (Danziger, 1990), null hypothesis statistical testing (NHST) has received consistent, largely unheeded criticism. Critiques have received more attention in recent years with the recognition of the replication crisis in the social sciences and the American Statistical Society's statement…

Cognitive Diagnosis Models (CDMs) are a special family of discrete latent variable models widely used in educational, psychological and social sciences. In many applications of CDMs, certain hierarchical structures among the latent attributes are assumed by researchers to characterize their dependence structure. Specifically, a directed acyclic…

Previous simulation studies of randomization tests applied in single-case educational intervention research contexts have typically focused on A-to-B phase changes in means/levels. In the present simulation study, we report the results of two multiple-baseline investigations, one targeting between-phase changes in slopes/trends and the other…

Researchers often have hypotheses concerning the state of affairs in the population from which they sampled their data to compare group means. The classical frequentist approach provides one way of carrying out hypothesis testing using ANOVA to state the null hypothesis that there is no difference in the means and proceed with multiple comparisons…

For ease of instruction in the classroom, the one-way analysis of variance F statistic is rewritten in terms of pairwise differences in individual sample means instead of differences of individual sample means from the overall sample mean. Likewise, the Kruskal-Wallis statistic may be rewritten in terms of pairwise differences in individual…

The 0.05 boundary within Null Hypothesis Statistical Testing (NHST) "has made a lot of people very angry and been widely regarded as a bad move" (to quote Douglas Adams). Here, we move past meta-scientific arguments and ask an empirical question: What is the psychological standing of the 0.05 boundary for statistical significance? We…

Multivariate analysis of variance (MANOVA) is widely used to test the null hypothesis of equal multivariate means across 2 or more groups. MANOVA rests upon an assumption that error terms are independent of one another, which can be violated if individuals are clustered or nested within groups, such as schools. Ignoring such nesting can result in…

This article studies the Type I error, false positive rates, and power of four versions of the Lagrange multiplier test to detect measurement noninvariance in item response theory (IRT) models for binary data under model misspecification. The tests considered are the Lagrange multiplier test computed with the Hessian and cross-product approach,…

Criteria are briefly proposed for final conclusions, research problems, and research hypotheses in quantitative research. Moreover, based on a proposed definition of applied and basic/general research, it is argued that (1) in applied quantitative research, while research problems are necessary, research hypotheses are unjustified, and that (2) in…

Meta-analytic models for dependent effect sizes have grown increasingly sophisticated over the last few decades, which has created challenges for a priori power calculations. We introduce power approximations for tests of average effect sizes based upon several common approaches for handling dependent effect sizes. In a Monte Carlo simulation, we…

Conducting research with human subjects can be difficult because of limited sample sizes and small empirical effects. We demonstrate that this problem can yield patterns of results that are practically indistinguishable from flipping a coin to determine the direction of treatment effects. We use this idea of random conclusions to establish a…

Research studies in Educational Data Mining (EDM) often involve several variables related to student learning activities. As such, it may be necessary to run multiple statistical tests simultaneously, thereby leading to the problem of multiple comparisons. The Benjamini-Hochberg (BH) procedure is commonly used in EDM research to address this…

In mixed methods approaches, statistical models are used to identify "nested" cases for intensive, small-n investigation for a range of purposes, including notably the examination of causal mechanisms. This article shows that under a commonsense interpretation of causal effects, large-n models allow no reliable conclusions about effect…