NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED677704
Record Type: Non-Journal
Publication Date: 2025-Oct-11
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
Between-Case Incidence Rate Ratio for Count Outcomes in Single Case Experimental Designs: A Monte Carlo Simulation with Conditional and Marginal Models
Haoran Li; Chendong Li; Wen Luo; Eunkyeng Baek
Society for Research on Educational Effectiveness
Background/Context: Single-case experiment designs (SCEDs) are experimental designs in which a small number of cases are repeatedly measured over time, with manipulation of baseline and intervention phases. Because SCEDs often rely on direct behavioral observations, count data are common. To account for both the clustering and the non-normal distribution of count data, generalized linear mixed models (GLMMs) have been shown as a promising approach (Declercq et al., 2019; Li et al., 2023; Shadish et al., 2013). In many fields, the incidence rate ratio (IRR) is the conventional choice for quantifying effect sizes in count outcomes, representing the proportional difference in incidence rates between the control and treatment conditions. In SCEDs, Li et al. (2023) proposed the within-case incidence rate ratio (WC-IRR) based on GLMMs, demonstrating its favorable statistical properties. The WC-IRR is a subject-specific effect size that allows for comparison and synthesis across SCEDs addressing similar research questions. However, it is not suitable for synthesis with effect size measures from between-group designs (e.g., RCTs). To bridge this gap, we recently developed the between-case incidence rate ratio (BC-IRR), a design-comparable version of the IRR for SCED count data (Li et al., 2025). Purpose/Objective/Research Questions: The purpose of this study is to evaluate the performance of GLMMs (conditional model) and generalized estimating equations (GEE, marginal model) regarding their accuracy and efficiency to estimate BC-IRR under various conditions. Definition and Estimation of BC-IRR: we used the following parametric model based on either a Poisson or a negative binominal GLMM for the data generation mechanism of SCED count data in an AB or a mutable baseline design. log([lambda][subscript ij]) = [beta][subscript 0j] + [beta][subscript 1j]Phase[subscript ij] + [beta][subscript 2j]Time[subscript ij] + [beta][subscript 3j]Time'[subscript ij]Phase[subscript ij] where Phase[subscript ij] is an indicator variable equal to 0 for control phase and 1 for treatment phase; Time[subscript ij] is a time variable representing the time (measured either in calendar units or sessions) elapsed from a reference time point. Time'[subscript ij] is another centered time variable which equals zero for the first session in the treatment phase. An example coding matrix for the variables is shown in Table 1. With a multivariate normal assumption for all random coefficients (i.e., [beta][subscript 0j], [beta][subscript 1j], [beta][subscript 2j], and [beta][subscript 3j]) [matrix omitted]. The BC-IRR = [equation omitted]. B-C represents the time elapsed between the reference time point C and the outcome measurement occasion B, while B-A represents the time elapsed between the introduction of treatment and the outcome measurement occasion. To compute BC-IRR, the estimates of the conditional effects ([gamma]) and variance/covariance components ([sigma]) from the GLMM are substituted into Equation 3. Alternatively, BC-IRR could be estimated by using a Poisson GEE model with an appropriate working correlation matrix: [equation omitted]. Based on Equation 4, BC-IRR = e[superscript ([gamma][subscript 20 superscript GEE]+[gamma][subscript 30 superscript GEE(B-A)]), where [gamma][subscript 20 superscript GEE] and [gamma][subscript 30 superscript GEE] represent the marginal estimates of the immediate effect and trend change, respectively. A key advantage of GEE over GLMM is that it does not require specifying the level-2 variance-covariance structure of random effects. However, in commonly used statistical software, GEE for count data is currently limited to the Poisson distribution. Simulation Design and Modeling: Data were simulated using a negative binomial GLMM. We varied the following design factors, including series length (I), number of cases (J), immediate effect ([gamma][subscript 20]), trend change ([gamma][subscript 30]), variance components ([sigma][subscript u0 superscript 2], [sigma][subscript u1 superscript 2], [sigma][subscript u2 superscript 2], [sigma][subscript u3 superscript 2]), and dispersion ([theta]). To calculate the true population values for BC-IRR, we considered time elapsed (B-A = 0, 1 or 5). Table 2 summarizes all design conditions. To analyze the simulated data, we fitted the negative binomial GLMM by using the R package "glmmTMB" (Brooks et al., 2017) and the GEE model via the R package "gee" (Carey, 2024). In GEE models, we applied two small-sample corrections for sandwich estimator standard errors: GEE_KC (Kauermann & Carroll, 2001) and GEE_MD (Mancl & DeRouen, 2001). Standard errors of the BC-IRRs for both models were then computed via the delta method using the R package "msm" (Jackson, 2011). Each model was estimated using two reference time points (C), namely, the first and last sessions of the baseline phase. To assess the statistical properties of GLMM and GEE for BC-IRR estimation, we examined bias in point and standard error estimates, mean squared error (MSE), and 95% confidence interval coverage rates. The full factorial design resulted in 864 conditions, each with 2,000 independent replications. Preliminary Findings: Preliminary results indicated that BC-IRR estimates exhibited low bias across all design factors. As shown in Table 3, both GLMM and GEE produced accurate estimates across different reference time points C and varying time intervals between intervention introduction time A and measurement time B. For standard error bias (Table 4), the two GEE corrections showed relative less bias when the reference time point C aligned with the simulated data (i.e., the last baseline session). However, bias increased when C was set to the first baseline session, particularly when B-A=5. In contrast, GLMMs exhibited consistently slight downward bias in standard errors across all conditions. MSE results (Table 5) indicated that GLMMs were generally more efficient for estimating BC-IRR than GEE models, particularly when the reference point C was set to the last baseline session. Coverage rate results (Table 6) showed that the GEE model with the MD correction and the GLMM outperformed the GEE model with the KC correction. Conclusion: This study systematically evaluated the statistical properties of BC-IRR estimates from GLMM and GEE through a Monte Carlo simulation. Results indicated that both models provided accurate estimates for BC-IRR, whereas GLMM is more robust to different reference time points C and the time lagged B-A, with generally better standard errors, MSE, and coverage rates than GEE. Our findings provide strong empirical support for using BC-IRR to compare and synthesize results with group-based designs such as RCTs. This enhances the ability of researchers and policymakers to identify evidence-based practices by integrating rigorous statistical findings from both SCEDs and RCTs.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A