As evidence from evaluation and experimental studies continue to influence decision and policymaking, applied researchers and practitioners require tools to derive valid and credible inferences. Over the past several decades, research in causal inference has progressed with the development and application of propensity scores. Since their…

Most existing solutions to the current replication crisis in science address only the factors stemming from specific poor research practices. We introduce a novel mechanism that leverages the experts' predictive abilities to analyze the root causes of replication failures. It is backed by the principle that the most accurate predictor is the most…

Missing data in scientific research go hand in hand with assumptions about the nature of the missingness. When dealing with missing values, a set of beliefs has to be formulated about the extent to which the observed data may also hold for the missing parts of the data. It is vital that the validity of these missingness assumptions is verified,…

Although recent studies have used the Classroom Observation Protocol for Undergraduate STEM (COPUS) to make claims about faculty reform, important questions remain: How should COPUS measures be situated within existing reform frameworks? Is there a universal sampling intensity that allows for valid inferences about the frequency of…

Multilevel modeling (MLM) is a statistical technique for analyzing clustered data. Despite its long history, the technique and accompanying computer programs are rapidly evolving. Given the complexity of multilevel models, it is crucial for researchers to provide complete and transparent descriptions of the data, statistical analyses, and results.…

The purpose of this study was to develop the Basic Literacy In Statistics (BLIS) assessment for students in an introductory statistics course, at the postsecondary level, that includes, to some extent, simulation-based methods. The definition of statistical literacy used in the development of the assessment was the ability to read, understand, and…

Data analysis is a significant methodological component when conducting quantitative education studies. Guidelines for conducting data analyses in quantitative education studies are common but often underemphasize four important methodological components impacting the validity of inferences: quality of constructed measures, proper handling of…

Fisher randomization tests for Neyman's null hypothesis of no average treatment effects are considered in a finite population setting associated with completely randomized experiments with more than two treatments. The consequences of using the F statistic to conduct such a test are examined both theoretically and computationally, and it is argued…

When randomized experiments are infeasible, quasi-experimental designs can be exploited to evaluate causal treatment effects. The strongest quasi-experimental designs for causal inference are regression discontinuity designs, instrumental variable designs, matching and propensity score designs, and comparative interrupted time series designs. This…

Online purposive samples have unknown biases and may not strictly be used to make inferences about wider populations, yet such inferences continue to occur. We compared the demographic and drug use characteristics of Australian ecstasy users from a probability (National Drug Strategy Household Survey, n = 726) and purposive sample (online survey…

In this study, the problem is limited factors relating to the learning effectiveness and teacher competence in improving the teacher performance. Therefore, this study will try to get explanations from some main issues which include the learning effectiveness issue, and teacher competence to increase teacher performance in Madrasah Tsanawiyah at…

An individual's neurodevelopmental and cognitive sequelae to negative early experiences may, in part, be explained by genetic susceptibility. We examined whether extreme differences in the early caregiving environment, defined as exposure to severe psychosocial deprivation associated with institutional care compared to normative rearing,…

This article introduces procedures for the computation and asymptotic statistical inference for classification consistency and accuracy indices specifically designed for cognitive diagnostic assessments. The new classification indices can be used as important indicators of the reliability and validity of classification results produced by…

As with any psychometric models, the validity of inferences from cognitive diagnosis models (CDMs) determines the extent to which these models can be useful. For inferences from CDMs to be valid, it is crucial that the fit of the model to the data is ascertained. Based on a simulation study, this study investigated the sensitivity of various fit…

The purpose of the present paper is to critically examine causal inferences and internal validity as defined by Campbell and co-workers. Several arguments are given against their counterfactual effect definition, and this effect definition should be considered inadequate for causal research in general. Moreover, their defined independence between…