Reweighting a distribution to minimize a distance to a target distribution is a powerful and flexible strategy for estimating a wide range of causal effects, but can be challenging in practice because optimal weights typically depend on knowledge of the underlying data generating process. In this paper, we focus on design-based weights, which do…

Randomized A/B tests allow causal estimation without confounding but are often under-powered. This paper uses a new dataset, including over 250 randomized comparisons conducted in an online learning platform, to illustrate a method combining data from A/B tests with log data from users who were not in the experiment. Inference remains exact and…

The Jeong (2020) study found that greater use of backward and depth-first processing was associated with higher scores on students' argument maps and that analysis of only the first five nodes students placed in their maps predicted map scores. This study utilized the jMAP tool and algorithms developed in the Jeong (2020) study to determine if the…

Students with mature number sense make sense of numbers and operations, use reasoning to notice patterns, and flexibly select the most effective and efficient problem-solving strategies (McIntosh et al., 1997; Yang, 2005). Despite being highlighted in national standards and policy documents (CCSS, 2010; NCTM, 2000), the association between…

Randomized A/B tests within online learning platforms enable us to draw unbiased causal estimators. However, precise estimates of treatment effects can be challenging due to minimal participation, resulting in underpowered A/B tests. Recent advancements indicate that leveraging auxiliary information from detailed logs and employing design-based…

This study examined the causal relationship of goal orientation and conception of learning on learning behavior; previously, these concepts were examined separately in preceding studies. A sample of 185 fourth-grade university students participated in the study. A covariance structure analysis revealed that the causal effect regarding goal…

Recent methods have sought to improve precision in randomized controlled trials (RCTs) by utilizing data from large observational datasets for covariate adjustment. For example, consider an RCT aimed at evaluating a new algebra curriculum, in which a few dozen schools are randomly assigned to treatment (new curriculum) or control (standard…

This paper provides the first mixed methods theory of causation. According to the theory, the researcher must carefully construct a causal mosaic for each research study, articulating what is causally relevant given his/her research questions, purposes, method(s), methodology(ies), paradigms(s), and resources. To engage in this "mixed…

The "doer effect" is the assertion that the amount of interactive practice activity a student engages in is much more predictive of learning than the amount of passive reading or watching video the same student engages in. Although the evidence for a doer effect is now substantial, the evidence for a causal doer effect is not as well…

Previous research has shown that providing video recommendations to students in virtual learning environments implemented at scale positively affects student achievement. However, it is also critical to evaluate whether the treatment effects are heterogeneous, and whether they depend on contextual variables such as disadvantaged student status and…

Randomized A/B tests in educational software are not run in a vacuum: often, reams of historical data are available alongside the data from a randomized trial. This paper proposes a method to use this historical data--often highdimensional and longitudinal--to improve causal estimates from A/B tests. The method proceeds in two steps: first, fit a…

Despite the prevalence of ordinal observed variables in applied structural equation modeling (SEM) research, limited attention has been given to model evaluation methods suitable for ordinal variables, thus providing practitioners in the field with few guidelines to follow. This study represents a first attempt to thoroughly examine the…

Researchers have highlighted how tracking learners' eye-gaze can reveal their reading behaviors and strategies, and this provides a framework for developing personalized feedback to improve learning and problem solving skills. In this paper, we describe analyses of eye-gaze data collected from 16 middle school students who worked with Betty's…

Classroom experiments that evaluate the effectiveness of educational technologies do not typically examine the effects of classroom contextual variables (e.g., out-of-software help-giving and external distractions). Yet these variables may influence students' instructional outcomes. In this paper, we introduce the Spatial Classroom Log Explorer…

The daily work of organising in schools both configures and is configured by the nature of schools as complex organisations. However, the organisational complexity of schools is often omitted in normative and analytical accounts of schools as organisations. In this paper, we seek to redress that omission and to bring organisational complexity to…