NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED677686
Record Type: Non-Journal
Publication Date: 2025-Oct-9
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
Two-Phase Treatment with Noncompliance: Identifying the Cumulative ATE via Multisite IV
Guanglei Hong; Xu Qin; Zhengyan Xu; Fan Yang
Society for Research on Educational Effectiveness
Background/Context: A well-designed intervention often involves multiple phases. A justifying theoretical argument is that, to prevent fade-outs, the benefit of a treatment in an earlier phase will need to be reinforced by follow-up treatments in the later phases (Cunha & Heckman, 2007). Thus, in evaluations of such programs, a research question of key scientific interest is the cumulative average treatment effect (ATE) of an intended experimental treatment sequence in contrast with a control treatment sequence for a target population. Even if the initial treatment assignment is randomized, some members of the target population who do not respond well to the Phase-I treatment may subsequently display noncompliant behaviors. In a multisite randomized trial, the compliance rate may vary across the sites. Identifying the cumulative ATE is challenging in the presence of noncompliance due to pre-existing measured and unmeasured differences between those who complied with the initial treatment assignment and those who did not. A naïve intention-to-treatment (ITT) analysis fails to adjust for such confounding. Inverse-probability-of-treatment weighting (IPTW) is suitable for handling observed pretreatment and posttreatment confounders but cannot remove unobserved confounding. When using the initial random treatment assignment as an instrumental variable (IV) to address noncompliance, the exclusion restriction is violated because the instrument is expected to exert a direct impact on the outcome through the Phase-I treatment regardless of whether one complies in Phase II. A multiple-site multiple-mediator IV strategy (Bloom et al, 2020; Duncan, Morris, & Rodrigues, 2011; Kling, Liebman, & Katz, 2007; Nomi & Raudenbush, 2021; Raudenbush, Reardon, & Nomi, 2012; Reardon and Raudenbush, 2013), developed for identifying the causal effects of multiple parallel mediators in a multisite trial, similarly requires the exclusion restriction and overlooks non-additive cumulative effects. Purpose/Objective/Research Question: We propose a multisite two-phase treatment IV (MS2T-IV) strategy to address noncompliance following the Phase-I treatment. Importantly, it does not invoke the overly strong assumption of ignorable Phase-2 treatment assignment given the observed covariates and thus is suitable for removing unmeasured pretreatment confounding. Moreover, this new strategy makes adjustment for observed posttreatment confounding and allows for non-additive treatment effects as well as the direct effect of the Phase-I treatment on the outcome, thereby relaxing the exclusion restriction. Setting/Intervention/Research Design: Authorized by the Tennessee state legislature in the 1980s, the Student Teacher Achievement Ratio study (Project STAR) was a state-wide multisite randomized trial designed to evaluate the cumulative impact of multi-year class size reduction on student achievement. Upon kindergarten entry, students within each of the nearly eighty participating schools were assigned at random to either a small class of 13-17 students (the experimental group) or a regular class of 22-25 students (the control group). Students were expected to remain in the initially assigned class size type throughout the study years. Teachers within each school were also assigned at random by the project staff to teach either a small class or a regular class at the beginning of each year (Finn & Achilles, 1990; Mosteller, Light, & Sachs, 1995). Yet student noncompliance started to occur at the beginning of Grade 1; the noncompliance rate was about 8% in each treatment group (Krueger, 1999). This study examines the cumulative ATE of receiving two years of instruction in a small class as opposed to a regular class in both kindergarten and Grade 1 on student achievement (the sum of reading and math test scores) by the end of Grade 1. The causal estimand ???????????????? is the average of the school-specific treatment effect for a population of schools. We also measure the sum of reading and math test scores by the end of kindergarten, which is a key posttreatment confounder in this study. Students who scored lower in kindergarten tend to have a higher likelihood of switching to the alternative class type in Grade 1. Identification and Estimation: Let Z[subscript ik] and D[subscript ik] denote the kindergarten treatment and Grade 1 treatment, respectively, for student i in school k. We specify theoretical models for the kindergarten potential outcome V[subscript ik](z), the Grade 1 potential treatment assignment D[subscript ik](z), and the Grade 1 potential outcome Y[subscript ik](z,d), based on which we derive the causal estimand [delta][subscript ATE] as a function of the theoretical model parameters. We then derive the reduced-form models for the observed V, D, and Y. They correspond to the analytic models in a stage-1 analysis for obtaining the school-specific ITT effects of Z on V, D, and Y. The stage-2 analysis is a school-level regression with the schoolspecific ITT effect of Z on Y as the outcome; the predictors include the school-specific ITT effects of Z on V and Z on D in addition to the school-specific compliance rate under the experimental condition. Let X and U denote observed and unobserved baseline covariates, respectively. Figure 1 illustrates the conceptual relationships. We clarify key identification assumptions that include Assumption 1 "Within-site ignorable initial treatment assignment," Assumption 2 "Within-site zero covariance assumption," and Assumption 3 "Between-site independence assumption." Assumption 1 is guaranteed by the multisite randomized design. However, Assumptions 2 and 3 can be violated. Potential consequences of violations can be assessed through a sensitivity analysis. Findings/Results/Conclusions: According to our simulation results, the proposed MS2T-IV method outperforms the IPTW method in bias reduction. Covariate adjustment for site-mean-centered baseline covariates in the Stage-1 ITT analyses further reduces finite sample bias. Although the MS2T-IV estimator is relatively less efficient in comparison with other methods, it shows the fastest rate in mean square error (MSE) reduction when the number of sites and the sample size per site increase. Applying the MS2T-IV strategy to the Project STAR data, we find that attending a small class as opposed to a regular class in both kindergarten and Grade 1 increases Grade 1 achievement by about 27% of a standard deviation. Inference based on multilevel bootstrap BCa (Efron & Tibshirani, 1993; Goldstein, 2011) or Zitzmann Jackknife (Zitzmann et al, 2023) suggests that this result is statistically significant. Future research may extend this strategy to evaluations of treatment sequences with more than two phases.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Early Childhood Education; Elementary Education; Kindergarten; Primary Education; Grade 1
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Tennessee
Grant or Contract Numbers: N/A
Author Affiliations: N/A