NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED677735
Record Type: Non-Journal
Publication Date: 2025-Oct-9
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
Experimental Power for Indirect Effects in Multisite Studies
Fangxing Bai; Ben Kelcey; Amota Ataneka; Yanli Xie; Kyle Cox; Nianbo Dong
Society for Research on Educational Effectiveness
Background: Multisite designs, also known as blocked designs, are experimental designs in which the random assignment of treatment and control conditions is within each site (or block) after the random selection of sites (or blocks). Multisite designs exhibit remarkable adaptability and, statistically, it can maintain a rigorous basis for inference. Consequently, these designs have been instrumental in evaluation studies across a broad range of fields. Multisite designs offer several key advantages. For instance, they enhance the efficiency of detecting ATEs by minimizing site-level confounding through the within-site assignment of treatment and control conditions. Furthermore, they facilitate the inclusion of diverse populations, providing opportunities to understand and explain effect heterogeneity across various groups (e.g., schools from different communities, locations, cultures, or demographic compositions). Additionally, by analyzing treatment effects across diverse contexts, multisite designs can strengthen the validity and improve the reproducibility of research studies (e.g., NICHD. 2023; Raudenbush & Bloom, 2015). Despite the widespread use of multisite designs, a notable gap exists in methodological guidance for multisite studies. Unlike other estimands, such as main or moderation effects, established methods for determining sample sizes required to detect indirect effects (key for studying the mechanisms of interventions) in multisite experiments are lacking. To address this gap, our study developed expressions to predict statistical power and optimal sampling plans for detecting mediation in multisite experiments. These results offer essential insights for enhancing the design efficiency and generalizability of findings in multisite mediation studies. Purpose: Multisite mediation studies are a cornerstone in mapping out developmental processes because they probe the mechanisms of a treatment while creating key opportunities to learn from and about variation in those mechanisms across sites. Despite the prevalence of multisite studies, a significant gap in the literature is how to plan such studies to detect indirect effects. Here, we develop formulas and an R "ShinyApp" to predict power and guide the design of multisite mediation studies. We developed expressions to predict statistical power for detecting mediation in multisite experiments. Our results are organized around two prevalent approaches in multisite studies: 1. Individual-level mediation (only): Focuses on settings where only individual-level effects are plausible (e.g., contextual influences are immaterial) and the proportion treated in a site is constant across sites. 2. Both individual contextual, and joint mediation: Considers settings where both individual and contextual effects are plausible and/or the proportion treated in a site varies across sites. Methods: We present the approach capturing both individual, contextual, and joint indirect effects for simplicity and will include both approaches in our fully developed paper. Prior studies have emphasized the importance of contextual effects--that is, site-level processes that operate in complement to individual-level changes (e.g., Pituch & Stapleton, 2012; Skinns et al., 2020). In the context of multisite studies, we can conceptually envision another form of treatment that contrasts substantively meaningful proportions of individuals within site who are exposed to the treatment. In turn, we can probe indirect effects originating from changes in the treatment assignment of an individual (holding constant proportion in site exposed) as well as indirect effects originating from changes in the proportion in a site exposed to treatment (holding constant individual treatment assignment). To address the complementary processes operating at the individual- and site-levels, we again draw on mixed effect models to estimate the conditional means (e.g., VanderWeele, 2010). We estimate the indirect effects using group-mean centering for individual-level predictors and introducing their means at the site level (e.g., Yaremych et al., 2023; Zhang et al. 2009). Under the group-mean centering specification, the model becomes [equations omitted]. Here we introduce [T-bar subscript J],[X-bar subscript J], and [M-bar subscript j] as the average of individual-level values within site j for T[subscript ij](treatment), X[subscript ij](covariate), and M[subscript ij](mediator), with their corresponding coefficients. Indirect effect. Individual-level mediation. Under the formulation, the indirect effect for a particular site is simply a[subscript j]b[subscript j]. However, because the a and b path coefficients can co-vary, the expected or average individual-level indirect effect (IIE) is IIE = E(a[subscript j]b[subscript j]) = ab + [tau subscript ab]. The resulting error variance of the expected value of the indirect effect is [equation omitted]. Here we use a and b as the maximum likelihood estimates of those paths, [sigma subscript ab] as the covariance between those paths with an error variance of [characters omitted], and [characters omitted] and [characters omitted] as the error variances. Contextual Indirect Effect. Similarly, we can assess the contextual indirect effect as CIE = AB - (ab + [tau subscript ab]). The CIE captures the difference in the potential outcomes through the mediator under exposure for two individuals who are connected to sites that differ by one unit in the proportion treated at a site. The error variance of the CIE can then be estimated using [equation omitted]. Joint Indirect Effect (JIE). We can also track the combined or JIE as it operates through changes in the individual-level treatment and mediator as well as changes in the proportion treated and the average mediator. The average JIE is E(AB) = AB. The expected error variance of the JIE can be traced as [equation omitted]. Illustration: Assume we would like to design a multi-school teacher-randomized study to assess how a teacher professional development program (T) enhances the quality of classroom instruction (Y) by improving teacher knowledge (M). We'll use an unbalanced design with varying treatment proportions within each site, considering site-level effects of aggregated treatment and mediator. Parameter values based on pilot studies are as follows: [equations omitted]. As in Figure 1, our power analyses suggested that the school sample size required to reach 80% power would be 130 for the joint indirect mediation and 165 for the individual-level moderated mediation. Summary: Mediation studies play a pivotal role in understanding program effects and advancing core theories. However, without robust statistical guidance, the inferences drawn from such studies may be limited. This can impede progress in evaluation science by constraining the breadth and reliability of evidence. In our study, we developed expressions to predict statistical power for detecting mediation in multisite studies, addressing this critical gap.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A