Propensity score matching and weighting methods are applied to balance covariates and reduce selection bias in the analysis of observational study data, and ultimately estimate a treatment effect. We wish to evaluate the impact of a Supplemental Instruction (SI) program on student success in an Introductory Statistics course. In such student…

Background: The closures resulting from the COVID-19 pandemic significantly impacted teacher licensure pathways. During that time, prospective teachers had less access to student teaching experiences, fewer program and course completion opportunities, and a reduced ability to prepare for and pass state licensing examinations. In response, a State…

This article reviews important statistical methods for estimating the impact of interventions on outcomes in education settings, particularly programs that are implemented in field, rather than laboratory, settings. We begin by describing the causal inference challenge for evaluating program effects. Then four research designs are discussed that…

Propensity and prognostic score methods are two statistical techniques used to correct for the selection bias in nonexperimental studies. Recently, the joint use of propensity and prognostic scores (i.e., two-score methods) has been proposed to improve the performance of adjustments using propensity or prognostic scores alone for bias reduction.…

At least sixteen US states have taken steps toward holding teacher preparation programs (TPPs) accountable for teacher value-added to student test scores. Yet it is unclear whether teacher quality differences between TPPs are large enough to make an accountability system worthwhile. Several statistical practices can make differences between TPPs…

Through a sensitivity analysis, the analyst attempts to determine whether a conclusion of causal inference could be easily reversed by a plausible violation of an identification assumption. Analytic conclusions that are harder to alter by such a violation are expected to add a higher value to scientific knowledge about causality. This article…

In the coming years, public programs will capture even more and richer data than they do now, including data from web-based tools used by participants in employment services, from tablet-based educational curricula, and from electronic health records for Medicaid beneficiaries. Program evaluators seeking to take full advantage of these data…

This brief provides a summary of the validity of Istation ISIP Early Reading scores in predicting students' performance levels on the Idaho Standards Achievement Test (ISAT) in English language arts (ELA). This correlational study analyzed how well third grade students' winter performance on the ISIP Early Reading test predicted their spring…

Researchers addressing posttreatment complications in randomized trials often turn to principal stratification to define relevant assumptions and quantities of interest. One approach for the subsequent estimation of causal effects in this framework is to use methods based on the "principal score," the conditional probability of belonging…

This case study examines the research process used to assess the intermediate effects of a community-based college preparation program in a fast growth, high needs exurban school district in Texas. Because it was not possible to randomly assign participants to treatment or control conditions, a nonexperimental design using propensity score…

Since school voucher funds are public, policymakers fiercely debate how those funds should be spent. A goal of many decision-makers is to ensure that every private school option is "high-quality" through program accountability regulations. Private schools, however, decide whether to participate in a private school choice program and…

There is growing pressure to make efficacy experiments more useful. This requires attending to the twin goals of generalizing experimental results to those schools that will use the results and testing the intervention's theory of action. We show how electronic records, created naturally during the daily operation of technology-based…

Objective: Training programs exist that prepare college students, faculty, and staff to identify and support students potentially at risk for suicide. Kognito is an online program that trains users through simulated interactions with virtual humans. This study evaluated Kognito's effectiveness in preparing users to intervene with at-risk students.…

Supplemental Instruction (SI) is a voluntary, non-remedial, peer-facilitated, course-specific intervention that has been widely demonstrated to increase student success, yet concerns persist regarding the biasing effects of disproportionate participation by already higher-performing students. With a focus on maintaining access for all students, a…

When implementing environmental education and interventions to promote one pro-environmental behavior, it is seldom asked if and how non-target pro-environmental behaviors are affected. The spillover effect proposes that engaging in one behavior affects the probability of engagement or disengaging in a second behavior. Therefore, the positive…