Applying qualitative comparative analysis (QCA) to large Ns relaxes researchers' case-based knowledge. This is problematic because causality in QCA is inferred from a dialogue between empirical, theoretical, and case-based knowledge. The lack of case-based knowledge may be remedied by various robustness tests. However, being a case-based method,…

Recent years have witnessed a host of innovations for conducting research with qualitative comparative analysis (QCA). Concurrently, important issues surrounding its uses have been highlighted. In this article, we seek to help users design QCA studies. We argue that establishing inference with QCA involves three intertwined design components:…

The internal validity of observational study is often subject to debate. In this study, we define the counterfactuals as the unobserved sample and intend to quantify its relationship with the null hypothesis statistical testing (NHST). We propose the probability of a robust inference for internal validity, that is, the PIV, as a robustness index…

As a research field geared toward understanding and improving learning, Learning Analytics (LA) must be able to provide empirical support for causal claims. However, as a highly applied field, tightly controlled randomized experiments are not always feasible nor desirable. Instead, researchers often rely on observational data, based on which they…

This article documents differences between novice and experienced undergraduate students' processes of reading mathematical proofs as revealed by moment-by-moment, think-aloud protocols. We found three key reading behaviors that describe how novices' reading differed from that of their experienced peers: alternative task models, accrual of…

Large-scale standardized tests are regularly used to measure student achievement overall and for student subgroups. These uses assume tests provide comparable measures of outcomes across student subgroups, but prior research suggests score comparisons across gender groups may be complicated by the type of test items used. This paper presents…

Motivation and self-concept count among the educationally most relevant factors and the evaluation of many educational interventions requires their valid measurement. The present study examined the psychometric properties of a shortened version of the Students' Approaches to Learning questionnaire measuring 10 distinct motivational and…

This report documents how one undergraduate student used set-based reasoning to reinvent logical principles related to conditional statements and their proofs. This learning occurred in a teaching experiment intended to foster abstraction of these logical relationships by comparing the predicate and inference structures among various proofs (in…

Quasi-experiments are common in studies that estimate the effect of instructional interventions on student performance outcomes. In this type of research, the nature of the experimental design, the choice in assessment, the selection of comparison groups, and the statistical methods used to analyze the comparison data dictate the validity of…

Although much research has compared the functioning between analytic and holistic rating scales, little research has compared the functioning of binary rating scales with other types of rating scales. This quantitative study set out to preliminarily and comparatively validate binary and analytic rating scales intended for use in formative…

Treatment effect estimates from a "regression discontinuity design" (RDD) have high internal validity. However, the arguments that support the design apply to a subpopulation that is narrower and usually different from the population of substantive interest in evaluation research. The disconnect between RDD population and the…

Among the multitude of studies that have examined an array of variables related to school leadership, only a small percentage have rigorously examined how school leaders spend their time. The complex role of school leaders poses challenges to common methods of collecting data about school leader time use, which subsequently threaten the validity…

A large body of research has examined students' conceptions of evolution and their relationships to acceptance of evolution. Proficiency in statistical and probabilistic reasoning has long been considered to be an essential feature of evolutionary reasoning, yet almost no empirical work has explored these putative connections. The RaPro…

This chapter introduces three applications of Bayesian inference to common and novel issues in second language research. After a review of the critiques of conventional hypothesis testing, our focus centers on ways Bayesian inference can be used for dealing with missing data, for testing theory-driven substantive hypotheses without a default null…

Most individuals with autism spectrum disorders often fail in tasks of theory of mind (ToM). However, those with normal intellectual functioning known as high functioning ASD (HF-ASD) sometimes succeed in mentalizing inferences. Some tools have been developed to more accurately test their ToM abilities. The aims of this study were to examine the…