Using a sample of 70,399 published p-values from 192 meta-analyses, we empirically estimate the counterfactual distribution of p-values in the absence of any biases. Comparing observed p-values with counterfactually expected p-values allows us to estimate how many p-values are published as being statistically significant when they should have been…

Designing and interpreting controlled experiments are important inquiry skills addressed in many current science curricula. The relevant skills associated with the design and interpretation of controlled experiments are summarised under the term control-of-variables strategy (CVS). Research on elementary school students' CVS skills shows that they…

Fetters et al.'s (2013) mixed methods integration framework uses construction metaphors: building, connecting, merging, and embedding. In a similar vein, this article uses an architectural metaphor and introduces design patterns as building blocks for mixed methods research design. A design pattern embeds one specific design decision into its…

This paper argues for three aspects of tolerance with respect to QCA research: tolerance with respect to different approaches to QCA; producing QCA research with tolerance (work that is resistant to criticism); and for QCA researchers to be clear about the tolerance of the solutions they present -- especially in terms of calibration and truth…

Researchers increasingly take advantage of the comparative case design to build theory, but the degree of case dependence is occasionally discussed and theorized. We suggest that the comparative case study design might be subject to an often underappreciated threat--dependence across cases--under certain conditions. Using research on innovation…

Background: Special education services are individualized supports provided to students with disabilities (SWD) that should help them make appropriate educational progress given their abilities, educational needs, and goals (Endrew F. v. Douglas County School District, 2017). Concerns about the benefits of these services have emerged given the…

This study aims to introduce network meta-analysis (NMA) to provide educational researchers with an extended view of the reviewing educational research. Meta-analytic methods have been widely used in educational research reviews. However, weaknesses have emerged in the multi-group comparison analysis of educational studies where different…

Graphene is a novel material with extraordinary promise in various nano-based applications, including increases in strength, conductivity, filtration, and electrical capacity. Graphene can be synthesized and tested in a wide variety of applications in secondary or undergraduate settings. Allowing students to choose their own application and method…

Recent years have witnessed a host of innovations for conducting research with qualitative comparative analysis (QCA). Concurrently, important issues surrounding its uses have been highlighted. In this article, we seek to help users design QCA studies. We argue that establishing inference with QCA involves three intertwined design components:…

Experimental studies are often employed to test the effectiveness of teaching innovations such as new pedagogy, curriculum, or learning resources. This article offers guidance on good practice in developing research designs, and in drawing conclusions from published reports. Random control trials potentially support the use of statistical…

This article develops new closed-form variance expressions for power analyses for commonly used difference-in-differences (DID) and comparative interrupted time series (CITS) panel data estimators. The main contribution is to incorporate variation in treatment timing into the analysis. The power formulas also account for other key design features…

As a research field geared toward understanding and improving learning, Learning Analytics (LA) must be able to provide empirical support for causal claims. However, as a highly applied field, tightly controlled randomized experiments are not always feasible nor desirable. Instead, researchers often rely on observational data, based on which they…

Treatment effect estimates from a "regression discontinuity design" (RDD) have high internal validity. However, the arguments that support the design apply to a subpopulation that is narrower and usually different from the population of substantive interest in evaluation research. The disconnect between RDD population and the…

Inquiry-based laboratories were implemented into a General Chemistry Laboratory sequence, and the impact of these exercises on students' experimental design skills was assessed using a four-part assessment developed for this study. This assessment contained a multiple-choice section, a section asking students to explain their reasoning behind a…

The regression discontinuity (RD) design is a popular quasi-experimental design for causal inference and policy evaluation. The most common inference approaches in RD designs employ "flexible" parametric and nonparametric local polynomial methods, which rely on extrapolation and large-sample approximations of conditional expectations…