Background: Over the past decade, research on the development and assessment of tools to improve the generalizability of experimental findings has grown extensively (Tipton & Olsen, 2018). However, many experimental studies in education are based on small samples, which may include 30-70 schools while inference populations to which…

Background/Context: The COVID-19 pandemic challenged the way we conduct research. For some modes of data collection, such as interviews, there was a ready (if not perfect) analog: face-to-face became phone-based; paper and pen surveys moved online. Others, such as direct assessments of child development, proved more challenging. Despite the…

Despite recent emphasis on the use of randomized control trials (RCTs) for evaluating education interventions, in most areas of education research, observational methods remain the dominant approach for assessing program effects. Over the last three decades, the within-study comparison (WSC) design has emerged as a method for evaluating the…

Difference-in-differences (DID) strategies are particularly useful for evaluating policy effects in natural experiments in which, for example, a policy affects some schools and students but not others. However, the standard DID method may produce biased estimation of the policy effect if the confounding effect of concurrent events varies by…

Randomized experiments are commonly used to evaluate if particular interventions improve student achievement. While these experiments can establish that a treatment actually "causes" changes, typically the participants are not randomly selected from a well-defined population and therefore the results do not readily generalize. Three…

Contradictory findings from "well-implemented" rigorous evaluations invite researchers to identify the differences that might explain the contradictions, helping to generate testable hypotheses for new research. This panel will examine efforts to ensure that the large number of local evaluations being conducted as part of four…

Propensity score analysis (PSA) is a methodological technique which may correct for selection bias in a quasi-experiment by modeling the selection process using observed covariates. Because logistic regression is well understood by researchers in a variety of fields and easy to implement in a number of popular software packages, it has…

The goal of this research is to make sensitivity analysis accessible not only to empirical researchers but also to the various stakeholders for whom educational evaluations are conducted. To do this it derives anchors for the omitted variable (OV)-program participation association intrinsically, using the Love plot to present a wide range of…

The purpose of this research to investigate the heterogeneity of per-pupil expenditure (PPE) slope estimates in predicting student achievement. The research question guiding this project is: how does the measured relationship between per-pupil expenditure vary across studies that use different models? In concert with SREE's 2012 conference mission…

Instrumental variable estimators hold the promise of enabling researchers to estimate the effects of educational treatments that are not (or cannot be) randomly assigned but that may be affected by randomly assigned interventions. Examples of the use of instrumental variables in such cases are increasingly common in educational and social science…

An educational intervention's effectiveness is judged by whether it produces positive outcomes for students, with the randomized controlled trial (CRT) as a valuable tool for determining intervention effects. However, the intervention-as-implemented in an experiment frequently differs from the intervention-as-designed, making it unclear whether…

This paper addresses methodological issues arising from an experimental study of North Carolina's Early College High School Initiative, a four-year longitudinal experimental study funded by Institute for Education Sciences. North Carolina implemented the Early College High School (ECHS) Initiative in response to low high school graduation rates.…