Formal empirical assessments of replication have recently become more prominent in several areas of science, including psychology. These assessments have used different statistical approaches to determine if a finding has been replicated. The purpose of this article is to provide several alternative conceptual frameworks that lead to different…

Structured means analysis is a very useful approach for testing hypotheses about population means on latent constructs. In such models, a z test is most commonly used for testing the statistical significance of the relevant parameter estimates or of the differences between parameter estimates, where a z value is computed based on the asymptotic…

White and Gorard make important and relevant criticisms of some of the methods commonly used in social science research, but go further by criticising the logical basis for inferential statistical tests. This paper comments briefly on matters we broadly agree on with them and more fully on matters where we disagree. We agree that too little…

Recent concerns about a shortage of capacity for statistical and numerical analysis skills among social science students and researchers have prompted a range of initiatives aiming to improve teaching in this area. However, these projects have rarely re-evaluated the content of what is taught to students and have instead focussed primarily on…

Quantitative results from empirical studies are common in the field of Scholarship of Teaching and Learning (SoTL), but it is important to remain aware of what the results from our studies can, and cannot, tell us. Oftentimes studies conducted to examine teaching and learning are constrained by class size. Small sample sizes negatively influence…

Null hypothesis significance testing (NHST) has been the subject of debate for decades and alternative approaches to data analysis have been proposed. This article addresses this debate from the perspective of scientific inquiry and inference. Inference is an inverse problem and application of statistical methods cannot reveal whether effects…

This study examined the extent to which school outdoor activities could enhance senior secondary (SS) two students' achievement in ecology. Non randomized pre test post test control group Quasi-experimental design was adopted. A sample of 160 SS II students from 4 co-educational schools in Jalingo metropolis, Taraba State Nigeria was used. A 40…

The motivation for this paper is the authors' recent work on several randomized control trials in which they found the primary result, which averaged across subgroups or sites, to be moderated by demographic or site characteristics. They are led to examine a distinction that the Institute of Education Sciences (IES) makes between "confirmatory"…

This paper examines strategies for interpreting and reporting estimates of intervention effects for subgroups of a study sample. Specifically, the paper considers: why and how subgroup findings are important for applied research, the importance of pre-specifying sub- groups before analyses are conducted, the importance of using existing theory and…

In his article, "An Alternative to Null-Hypothesis Significance Tests," Killeen (2005) urged the discipline to abandon the practice of "p[subscript obs]"-based null hypothesis testing and to quantify the signal-to-noise characteristics of experimental outcomes with replication probabilities. He described the coefficient that he…

The aim of the current study is to validate the invariance of major profile patterns derived from multidimensional scaling (MDS) by bootstrapping. Profile Analysis via Multidimensional Scaling (PAMS) was employed to obtain profiles and bootstrapping was used to construct the sampling distributions of the profile coordinates and the empirical…

Assessing the comparability of different groups is an issue facing many researchers and evaluators in a variety of settings. Commonly, null hypothesis significance testing (NHST) is incorrectly used to demonstrate comparability when a non-significant result is found. This is problematic because a failure to find a difference between groups is not…

An experiment compared asynchronous versus synchronous instruction in an online quantitative course. Mann-Whitney U-tests, correlation, analysis of variance, t tests, and multivariate analysis of covariance (MANCOVA) were utilized to test the hypothesis that more high-quality online experiential learning interactions would increase grade.…

The sense that replicability is an important aspect of empirical science led Killeen (2005a) to define "p[subscript rep]," the probability that a replication will result in an outcome in the same direction as that found in a current experiment. Since then, several authors have praised and criticized 'p[subscript rep]," culminating…

This report presents guidelines for addressing the multiple comparisons problem in impact evaluations in the education area. The problem occurs due to the large number of hypothesis tests that are typically conducted across outcomes and subgroups in these studies, which can lead to spurious statistically significant impact findings. The…