Reweighting a distribution to minimize a distance to a target distribution is a powerful and flexible strategy for estimating a wide range of causal effects, but can be challenging in practice because optimal weights typically depend on knowledge of the underlying data generating process. In this paper, we focus on design-based weights, which do…

Randomized A/B tests allow causal estimation without confounding but are often under-powered. This paper uses a new dataset, including over 250 randomized comparisons conducted in an online learning platform, to illustrate a method combining data from A/B tests with log data from users who were not in the experiment. Inference remains exact and…

Randomized A/B tests within online learning platforms enable us to draw unbiased causal estimators. However, precise estimates of treatment effects can be challenging due to minimal participation, resulting in underpowered A/B tests. Recent advancements indicate that leveraging auxiliary information from detailed logs and employing design-based…

Random sampling and random allocation are essential processes in the practice of inferential statistics. These processes ensure that all members of a population are equally likely to be selected, and that all possible allocations in an experiment are equally likely. It is these characteristics that allow the validity of the subsequent calculations…

In this paper, we address the following questions: What misconceptions do high school students exhibit in their first encounter with significance test problems through a repeated sampling approach? Which theory or framework could explain the presence and features of such patterns? With brief prior instruction on the use of Fathom software to…

The analysis of variance (ANOVA) F test is a commonly used method to test the mean equality among two or more populations. A critical assumption of ANOVA is homogeneity of variance (HOV), that is, the compared groups have equal variances. Although it is encouraged to test HOV as part of the regular ANOVA procedure, the efficacy of the initial HOV…

Single-case design is a repeated-measures research approach for the study of the effect of an intervention, and its importance is increasingly being recognized in education and psychology. We propose a Bayesian approach for estimating intervention effects in SCD. A Bayesian inference does not rely on large sample theories and thus is particularly…

It is well known that students of inferential statistics find the hypothetical, probabilistic reasoning used in hypothesis tests difficult to understand. Consequently, they will also have difficulties in understanding "p"-values. It is not unusual for these students to hold misconceptions about "p"-values that are difficult to…

In the present study we analyze how students reason about or make inferences given a particular hypothesis testing problem (without having studied formal methods of statistical inference) when using Fathom. They use Fathom to create an empirical sampling distribution through computer simulation. It is found that most student´s reasoning rely on…

This study focuses on the development of four tertiary introductory statistics students' informal inferential reasoning while engaging in data driven repeated sampling and resampling activities. Through the use of hands-on manipulatives and simulations with technology, the participants constructed empirical sampling distributions in order to…

An understanding of conditional probability is essential for students of inferential statistics as it is used in Null Hypothesis Tests. Conditional probability is also used in Bayes' theorem, in the interpretation of medical screening tests and in quality control procedures. This study examines the understanding of conditional probability of…

Exploratory factor analysis involves five key decisions. The second decision, how many factors to retain, is the focus of the current paper. Extracting too many or too few factors often leads to devastating effects on study results. The advantages and disadvantages of the most effective and/or most utilized strategies to determine the number of…

This paper explores the impact on teachers' self-efficacy to teach statistics from a graduate course aimed to develop teachers' knowledge of inferential statistics through engaging in data analysis using technology. This study uses qualitative and quantitative data from the Self-Efficacy to Teach Statistics Survey (Harrell-Williams et al., 2013)…

As result replicability is essential to science and difficult to achieve through external replicability, the present paper notes the insufficiency of null hypothesis statistical significance testing (NHSST) and explains the bootstrap as a plausible alternative, with a heuristic example to illustrate the bootstrap method. The bootstrap relies on…

This paper examines the tension in the process of designing student surveys between the methodological requirements of good survey design and the institutional needs for survey data. Building on the commonly used argumentative approach to construct validity, I build an interpretive argument for student opinion surveys that allows assessment of the…