This article presents two alternative methods to null hypothesis significance testing (NHST) for improving inferences from underpowered research designs. Post hoc design analysis (PHDA) assesses whether an NHST analysis generating null findings might otherwise have had sufficient power to detect effects of plausible magnitudes. Bayesian analysis…

Students often believe that statistical significance is the only determinant of whether a quantitative result is "important." In this paper, I review traditional null hypothesis statistical testing to identify what questions inferential statistics can and cannot answer, including statistical significance, effect size and direction,…

This comment discusses some issues about the use and reporting of experimental studies in education, illustrated by a recently published study that claimed (i) that an educational innovation was effective despite outcomes not reaching statistical significance, and (ii) that this refuted the findings of an earlier study. The two key issues raised…

The purpose of this quantitative non-experimental study is to determine the leadership competencies most desired by members of the community college board of trustees and faculty for a community college president. The study will examine the alignment of those competencies between the trustees and their faculty groups to determine if the two groups…

Recently, Stephen Gorard has outlined strong objections to the use of significance testing in social research. He has argued, first, that as the samples used in social research are almost always non-random it is not possible to use inferential statistical techniques and, second, that even if a truly random sample were achieved, the logic behind…

This monograph provides an epistemological rational for the design of a novel post hoc statistical measure called "Tri-Center Analysis". This new statistic is designed to analyze the post hoc outcomes of the Tri-Squared Test. In Tri-Center Analysis trichotomous parametric inferential parametric statistical measures are calculated from…

Purpose: Four purposes were present in this journal-ready dissertation. The first purpose was to examine the degree to which the percentage of beginning teachers in traditionally configured elementary, middle, and secondary Texas schools differed as a function of two distinction designations (i.e., Academic Achievement in Reading/English Language…

I discuss three common practices that obfuscate or invalidate the statistical analysis of randomized controlled interventions in applied linguistics. These are (a) checking whether randomization produced groups that are balanced on a number of possibly relevant covariates, (b) using repeated measures ANOVA to analyze pretest-posttest designs, and…

Many librarians use data from surveys to make decisions about how to spend money or allocate staff, often making use of popular online tools like Survey Monkey. In this era of reduced budgets, low staffing, stiff competition for new resources, and increasingly complex choices, it is especially important that librarians know how to get strong,…

Purpose: Researchers and scholars have called for greater attention to collaboration among and between educational leaders in districtwide reform. This work underlines the important social aspect of such collaboration and further investigates the type of professional interaction among/between district and school leaders particularly around the…

We surveyed researchers in the health and exercise sciences to explore different areas and magnitudes of bias in researchers' decision making. Participants were presented with scenarios (testing a central hypothesis with p = 0.06 or p = 0.04) in a random order and surveyed about what they would do in each scenario. Participants showed significant…

The use of confidence intervals (CIs) as an addition or as an alternative to null hypothesis significance testing (NHST) has been promoted as a means to make researchers more aware of the uncertainty that is inherent in statistical inference. Little is known, however, about whether presenting results via CIs affects how readers judge the…

The purpose of the current study is to investigate the effects of frequency of peer victimization experiences on psychological and academic adjustment during early adolescence, with a focus on testing psychological adjustment as a mediator, as well as differences based on gender and type of victimization. The sample in this short-term longitudinal…

This "What Works Clearinghouse Procedures and Standards Handbook (Version 3.0)" provides a detailed description of the standards and procedures of the What Works Clearinghouse (WWC). The remaining chapters of this Handbook are organized to take the reader through the basic steps that the WWC uses to develop a review protocol, identify…

Null hypothesis significance testing has dominated quantitative research in education and psychology. However, the statistical significance of a test as indicated by a p-value does not speak to the practical significance of the study. Thus, reporting effect size to supplement p-value is highly recommended by scholars, journal editors, and academic…