Caregivers of children with attention deficit-hyperactivity disorder (ADHD) and autism spectrum disorder (ASD) experience more stress than caregivers of typically developing children but there is limited research evaluating caregivers' quality of life (QoL). This study aimed to describe the association of caregiver QoL in children with ASD and/or ADHD. This study included patients with ADHD and/or ASD seen in one pediatric specialty clinic between September 2018-August 2020. Caregivers were classified as those caring for children with ASD-only, ADHD-only, or youth with both conditions (ADHD + ASD). An adapted version of the PedsQL Family Impact Module was used to measure caregiver QoL. The sample included caregivers of 931 children. The majority of these children were male (74.7%), non-Hispanic white (63.3%), and aged 6 to 12 years (57.8%). Across the groups, significant differences were observed in patient age (p < 0.0001), preferred language (p = 0.005), and insurance (p = 0.001). Caregivers of non-Hispanic Black children had 4-times the odds of reporting feeling isolated from others (OR 4.36, 95% CI 1.19-16.00 p = 0.03). Those caring for children with ADHD-only had significantly lower odds of reporting helplessness or hopelessness (OR 0.45, 95% CI 0.26-0.80, p = 0.004), and difficulty talking about their child's health with others (OR 0.30, 95% CI 0.17-0.54, p < 0.0001). Similarly, caregivers of children who had ADHD + ASD reported higher odds of difficulty making decisions together as a family (OR 14.18, 95% CI 1.15-17.91, p=0.04) and difficulty solving family problems together (OR 45.12, 95% CI 2.70-752.87), p = 0.008). Caring for children with ADHD and/or ASD may affect caregiver QoL.