This paper argues for three aspects of tolerance with respect to QCA research: tolerance with respect to different approaches to QCA; producing QCA research with tolerance (work that is resistant to criticism); and for QCA researchers to be clear about the tolerance of the solutions they present -- especially in terms of calibration and truth…

The use of Mixed Methods (MM) in evaluation has increasingly become a valuable asset in evaluators' methodological toolbox. Despite prolific research on various aspects of mixed methods, from typologies to ontological debates, there remains a dearth of empirically derived evidence on effectively communicating MM findings. This lack of empirical…

In the last few years, Qualitative Comparative Analysis (QCA) has become one of the most important data analysis methods in comparative research. According to the guidelines of this method, there are certain steps that a researcher needs to follow, before causally analyzing the data for necessary and sufficient conditions. One of these steps is…

Researchers in second language (L2) and education domain use different statistical methods to assess their constructs of interest. Many L2 constructs emerge from elements/parts, i.e., the elements "define" and "form" the construct and not the other way around. These constructs are referred to as emergent variables (also called…

A small subset of education studies analyzes school data collected seasonally (separating the summer from the school year). At first, this work was primarily known for documenting learning loss in the summers, but scholars have since recognized that observing how inequality changes between summer and school periods provides leverage for…

We develop a structural after measurement (SAM) method for structural equation models (SEMs) that accommodates missing data. The results show that the proposed SAM missing data estimator outperforms conventional full information (FI) estimators in terms of convergence, bias, and root-mean-square-error in small-to-moderate samples or large samples…

As a research field geared toward understanding and improving learning, Learning Analytics (LA) must be able to provide empirical support for causal claims. However, as a highly applied field, tightly controlled randomized experiments are not always feasible nor desirable. Instead, researchers often rely on observational data, based on which they…

Tests of statistical significance often play a decisive role in establishing the empirical warrant of evidence-based research in education. The results from pattern-based assessment items, as introduced in this paper, are categorical and multimodal and do not immediately support the use of measures of central tendency as typically related to…

Adaptive Comparative Judgment (ACJ), an approach to the assessment of open-ended problems which utilizes a series of comparisons to produce a standardized score, rank order, and a variety of other statistical measures, has demonstrated high levels of reliability and validity and the potential for application in a wide variety of areas. Further,…

Factor analysis is widely utilized to identify latent factors underlying the observed variables. This paper presents a comprehensive comparative study of two widely used methods for determining the optimal number of factors in factor analysis, the K1 rule, and parallel analysis, along with a more recently developed method, the bass-ackward method.…

Network meta-analysis (NMA) incorporates all available evidence into a general statistical framework for comparing multiple treatments. Standard NMAs make three major assumptions, namely homogeneity, similarity, and consistency, and violating these assumptions threatens an NMA's validity. In this article, we suggest a graphical approach to…

Component network meta-analysis (CNMA) compares treatments comprising multiple components and estimates the effects of individual components. For network meta-analysis, a standard network plot with nodes for treatments and edges for direct comparisons between treatments is drawn to visualize the evidence structure and the connections between…

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two…

In recent years, proponents of configurational comparative methods (CCMs) have advanced various dimensions of robustness as instrumental to model selection. But these robustness considerations have not led to computable robustness measures, and they have typically been applied to the analysis of real-life data with unknown underlying causal…

The What Works Clearinghouse (WWC) is an initiative of the U.S. Department of Education's Institute of Education Sciences (IES), which was established under the Education Sciences Reform Act of 2002. It is an important part of IES's strategy to use rigorous and relevant research, evaluation, and statistics to improve the nation's education system.…