Several programs of research have sought to assess the replicability of scientific findings in different fields, including economics and psychology. These programs attempt to replicate several findings and use the results to say something about large-scale patterns of replicability in a field. However, little work has been done to understand the…

This paper examines sources of potential bias in systematic reviews and meta-analyses which can distort their findings, leading to problems with interpretation and application by practitioners and policymakers. It follows from an article that was published in the "Canadian Journal of Communication" in 1990, "Integrating Research…

Overlap-based measures are increasingly applied in the synthesis of single-subject research. This article considers two criticisms of overlap-based metrics, specifically that they do not measure magnitude of effect and do not adequately correspond with visual analysis. It is argued that these criticisms are based on fundamental misconceptions…

In this article, the authors address the practicalities of how data should be presented, summarized, and interpreted. There are no exact rules; indeed there are valid concerns that exact rules may be inappropriate and too prescriptive. New procedures evolve, and new methods may be needed to deal with new types of data, just as people know that new…

Confidence intervals (CIs) and effect sizes are essential to encourage meta-analytic thinking and to accumulate research findings. CIs provide a range of plausible values for population parameters with a degree of confidence that the parameter is in that particular interval. CIs also give information about how precise the estimates are. Comparison…

The Journal of Agricultural Education (JAE) requires authors to follow the guidelines stated in the Publication Manual of the American Psychological Association [APA] (2009) in preparing research manuscripts, and to utilize accepted research and statistical methods in conducting quantitative research studies. The APA recommends the reporting of…

The National Early Literacy Panel (2008) report identified early predictors of reading achievement as good targets for instruction, and many of those skills are related to decoding. In this article, the authors suggest that the developmental trajectories of rapidly developing skills pose problems for traditional statistical analyses. Rapidly…

The authors critiqued the M. K. Lovelace (2005) meta-analysis of the Dunn and Dunn Model of Learning-Style Preferences (DDMLSP). The conclusion that Lovelace reported in her meta-analysis that learning-style instruction is a beneficial form of instructional delivery is unjustified because of critical conceptual and practical problems. Those…

The present article presents a tutorial on how to estimate and interpret various effect sizes. The 5th edition of the Publication Manual of the American Psychological Association (2001) described the failure to report effect sizes as a "defect" (p. 5), and 23 journals have published author guidelines requiring effect size reporting. Although…

Why methodologists encourage the use of magnitude-of-effect (ME) indices as research interpretation aids is discussed, and different types of ME estimates are reviewed. Correction formulas developed to alternate statistical bias in ME estimates are also discussed, and their effects are illustrated. (SLD)

Problems inherent in relying solely on statistical significance testing as a means of data interpretation are reviewed. The biggest problem with statistical significance testing is that researchers have used the results of this testing to ascribe importance or meaning to their studies where such meaning often does not exist. Often researchers…

This paper examines the problems caused by relying solely on statistical significance tests to interpret results in contemporary social science. The place of significance testing in educational research has often been debated. Among the problems in reporting statistical significance are questions of definition and terminology. Problems are also…

This paper details the challenges encountered by authors summarizing evidence from a primary study to describe a treatment's effectiveness using an effect size (ES) estimate. Dilemmas that are encountered, including how to calculate and interpret the pertinent standardized mean difference ES for results from studies of various research designs,…

Problems with using stepwise analytic methods are discussed, and better alternatives are illustrated. To make the illustrations concrete, an actual data set, involving responses of 91 medical school admissions directors to 30 variables, was used. The 30 variables involved perceptions of barriers to medical school with respect to characteristics of…

Magnitude of effect measures (MEMs), when adequately understood and correctly used, are important aids for researchers who do not want to rely solely on tests of statistical significance in substantive result interpretation. The MEM tells how much of the dependent variable can be controlled, predicted, or explained by the independent variables.…