Single-case experimental designs (SCEDs) have a long history in clinical and educational disciplines. One underdeveloped area in advancing SCED design and analysis is understanding the process of how internal validity threats and operational concerns are avoided or mitigated. Two strategies to ameliorate such issues in SCED involve replication and…

The randomized control trial (RCT) is the primary experimental design in education research due to its strong internal validity for causal inference. However, in situations where RCTs are not feasible or ethical, quasi-experiments are alternatives to establish causal inference. This paper serves as an introduction to several quasi-experimental…

This guide offers a comprehensive handbook to scientific research methodology and experimental design, specifically for novice MA and PhD researchers in Education and Language Learning (TESOL/TEFL). It establishes scientific research as a systematic, objective inquiry focused on identifying cause-and-effect relationships through empirical data.…

Despite calls for more evidence regarding the effectiveness of teacher education practices, causal research in the field remains rare. One reason is that we lack designs and measurement approaches that appropriately meet the challenges of causal inference in the context of teacher education programs. This article provides a framework for how to…

In science in general and in the context of single-case experimental designs, replication of the effects of the intervention within and/or across participants or experiments is crucial for establishing causality and for assessing the generality of the intervention effect. Specific developments and proposals for assessing whether an effect has been…

As a research field geared toward understanding and improving learning, Learning Analytics (LA) must be able to provide empirical support for causal claims. However, as a highly applied field, tightly controlled randomized experiments are not always feasible nor desirable. Instead, researchers often rely on observational data, based on which they…

Students often believe that statistical significance is the only determinant of whether a quantitative result is "important." In this paper, I review traditional null hypothesis statistical testing to identify what questions inferential statistics can and cannot answer, including statistical significance, effect size and direction,…

Treatment effect estimates from a "regression discontinuity design" (RDD) have high internal validity. However, the arguments that support the design apply to a subpopulation that is narrower and usually different from the population of substantive interest in evaluation research. The disconnect between RDD population and the…

In policy evaluations, interest may focus on why a particular treatment works. One tool for understanding why treatments work is causal mediation analysis. In this essay, I focus on the assumptions needed to estimate mediation effects. I show that there is no "gold standard" method for the identification of causal mediation effects. In…

Single case studies can provide vital contributions to theory-testing in social science studies. Particularly, by applying the process-tracing method, case studies can test theoretical frameworks through a rigorous research design that ensures substantial empirical leverage. While most scholarly contributions on process-tracing focus on either…

Design research is strongly associated with the learning sciences community, and in the 2 decades since its conception it has become broadly accepted. Yet within and without the learning sciences there remains confusion about how to do design research, with most scholarship on the approach describing what it is rather than how to do it. This…

The reaction that occurs when Mentos are added to bottled soft drinks has become a staple demonstration in earth science courses to explain how volcanoes erupt. This paper presents how this engaging exercise can be used in a marketing research course to provide hands-on experience with problem formation, hypothesis testing, and causal research. A…

Legitimate knowledge claims about causation have been a central concern among evaluators and applied researchers for several decades and often have been the subject of heated debates. In recent years these debates have resurfaced with a renewed intensity, due in part to the priority currently being given to randomized experiments by many funders…

This article reports on a delicious finding from a recent study claiming a causal link between dark chocolate consumption and blood pressure reductions. In the article, I provide ideas for using this study to whet student appetites for a discussion of statistical ideas, including experimental design, measurement error and inference methods.

Mediation and moderation are two theories for refining and understanding a causal relationship. Empirical investigation of mediators and moderators requires an integrated research design rather than the data analyses driven approach often seen in the literature. This paper described the conceptual foundation, research design, data analysis, as…