Background: Over the past decade, research on the development and assessment of tools to improve the generalizability of experimental findings has grown extensively (Tipton & Olsen, 2018). However, many experimental studies in education are based on small samples, which may include 30-70 schools while inference populations to which…

This comment discusses some issues about the use and reporting of experimental studies in education, illustrated by a recently published study that claimed (i) that an educational innovation was effective despite outcomes not reaching statistical significance, and (ii) that this refuted the findings of an earlier study. The two key issues raised…

The ability of a scientific discipline to build cumulative knowledge depends on its predominant method of data analysis. A steady accumulation of knowledge requires approaches which allow researchers to consider results from comparable prior research. Bayesian statistics is especially relevant for establishing a cumulative scientific discipline,…

The central purpose of this article is to provide an overview of the many ways in which special educators can generate and think about causal inference to inform policy and practice. Consideration of causality across different lenses can be carried out by engaging in multiple method and mixed methods ways of thinking about inference. This article…

Multilevel models are often used to evaluate hypotheses about relations among constructs when data are nested within clusters (Raudenbush & Bryk, 2002), although alternative approaches are available when analyzing nested data (Binder & Roberts, 2003; Sterba, 2009). The overarching goal of this article is to suggest when it is appropriate…

The authors thank Dr. Lindsay Page for providing a nice illustration of the use of the principal stratification framework to define causal effects, and a Bayesian model for effect estimation. They hope that her well-written article will help expose education researchers to these concepts and methods, and move the field of mediation analysis in…

Nearly all quantitative analyses in higher education draw from incomplete datasets-a common problem with no universal solution. In the first part of this paper, we explain why missing data matter and outline the advantages and disadvantages of six common methods for handling missing data. Next, we analyze real-world data from 5,905 students across…

Professors Richard J. Murnane and John B. Willett set out to capitalize on recent developments in education data and methodology by attempting to answer the following questions: How can new methods and data be applied most effectively in educational and social science research? What kinds of research designs are most appropriate? What kinds of…

This is the first report from a new initiative called TLRI Project Plus. It aims to add value to the Teaching and Learning Research Initiative (TLRI), which NZCER manages on behalf of the government, by synthesising findings across multiple projects. This report focuses on two projects in statistics education and explores the factors that…

In the wake of an increasing political commitment to evidence-based decision making and evidence-based educational reform that emerged with the No Child Left Behind effort, the question of what counts as evidence has become increasingly important in the field of science education. In current public discussions, academics, politicians, and other…

This report examines the differences in impact estimates and standard errors that arise when these are derived using state achievement tests only (as pre-tests and post-tests), study-administered tests only, or some combination of state- and study-administered tests. State tests may yield different evaluation results relative to a test that is…

Hierarchical models are extremely promising tools for data analysis, but it is important not to lessen hard thinking about data and iterative model checking when fitting hierarchical models. More and better software, methods to assure proper calibration, and materials in support of hierarchical model use are all needed. (SLD)

Describes the logic of statistical testing in the Fisher and Neyman-Pearson approaches. Reviews some common misinterpretations of basic concepts behind statistical tests. Analyzes the philosophical and psychological issues that can contribute to these misinterpretations. Suggests possible ways in which statistical education might contribute to the…

Bootstrap analysis, both for nonparametric statistical inference and for describing sample results stability and replicability, has been gaining prominence among quantitative researchers in educational and psychological research. Procedurally, however, it is often quite a challenge for quantitative researchers to implement bootstrap analysis in…

Sixty-seven educational research journals were investigated to determine the frequency of usage of inferential statistical techniques therein. The most frequently used statistical methodologies in the literature reviewed, which utilized inferential approaches, are the following: analysis of variance, correlation, t-test, multiple analysis of…