Using a sample of 70,399 published p-values from 192 meta-analyses, we empirically estimate the counterfactual distribution of p-values in the absence of any biases. Comparing observed p-values with counterfactually expected p-values allows us to estimate how many p-values are published as being statistically significant when they should have been…

The design of applied linguistic interventions to deal with large-scale language problems involves processes that are both intentional and deliberate. Though they range from the highly innovative to the conventional, applied linguistic designs are not necessarily disruptive. The design process is characterized by five phases, with possible…

This article reviews the origins and use of the terms quasi-experiment and natural experiment. It demonstrates how the terms conflate whether variation in the independent variable of interest falls short of random with whether researchers find, rather than intervene to create, that variation. Using the lens of assignment--the process driving…

Within educational research, qualitative data offers unique opportunities to contextualize findings that arise from large-scale quantitative data collection processes. Still, to date, most educational research is based on researchers' determination of what is important, not the participants'. Participatory Rank Methodology (PRM) is a…

The five SEERNet digital learning platforms (DLPs) present unique opportunities for researchers by offering tools, processes, and infrastructure to make research more efficient, scalable, and relevant. However, conducting research within a DLP may require a shift in a researcher's orientation or mindset in how they think about potential research…

In a clustered observational study, a treatment is assigned to groups and all units within the group are exposed to the treatment. We develop a new method for statistical adjustment in clustered observational studies using approximate balancing weights, a generalization of inverse propensity score weights that solve a convex optimization problem…

Mediation analyses are crucial for understanding the mechanisms through which interventions or theoretical constructs influence outcomes within nested structures, commonly found in education, psychology, and sociology. Recognizing the importance of these effects, designing and analyzing robust studies to detect them is essential. This dissertation…

Single-case design (SCD) is a quantitative experimental technique in which participants serve as their own control. The use of an effect size in SCD allows evaluation of outcomes as well as comparison of outcomes via meta-analyses. Characteristics of SCD research make the selection of an appropriate effect size complicated. Additionally, there are…

Single-case design (SCD) is a quantitative experimental technique in which participants serve as their own control. The use of an effect size in SCD allows evaluation of outcomes as well as comparison of outcomes via meta-analyses. Characteristics of SCD research make the selection of an appropriate effect size complicated. Additionally, there are…

Until now, the conventional approach using two distinct groups, experimental and control, continues to dominate research, especially education research. Researchers, particularly those who are active in this domain, readily recognise this pattern when surveying literature. This article explores the use of the Solomon four-group design as a…

Conducting survey research in rural educational contexts presents unique challenges and requires specific methodological considerations. This article provides an overview of potential adaptations that may help support effective, valid, and impactful rural survey research and a brief review of survey research published in "Rural Special…

This systematic literature review examines how experimental design is taught in chemistry education by using the PRISMA 2020 methodology. The review aims to identify research trends, analyze pedagogical approaches, and examine the pedagogical strategies and methods employed in this process, particularly concerning the preparation of future…

Medical students must understand statistical reasoning and sample size selection to design and interpret clinical trials. Beyond achieving sufficient statistical power, ensuring meaningful precision in treatment effect estimates is equally important. We developed free, interactive Shiny/R tools that let learners explore how varying sample sizes…

A key aspect of metacognition is metacognitive accuracy, i.e., the degree to which confidence judgments differentiate between correct and incorrect trials. To quantify metacognitive accuracy, researchers are faced with an increasing number of different methods. The present study investigated false positive rates associated with various measures of…

Systematic reviews have been gaining attention as a research methodology and are among the most frequently cited sources in educational sciences. However, best practices for conducting systematic reviews in educational sciences are still evolving. We conducted N = 12 qualitative interviews to learn from experienced systematic review researchers…