Reading disorders are common in children and can impact academic success, mental health, and career prospects. Reading is supported by network of interconnected left hemisphere brain regions, including temporo-parietal, occipito-temporal, and inferior-frontal circuits. Poor readers often show hypoactivation and reduced gray matter volumes in this reading network, with hyperactivation and increased volumes in the posterior right hemisphere. We assessed gray matter development longitudinally in pre-reading children aged 2-5 years using magnetic resonance imaging (MRI) (N = 32, 110 MRI scans; mean age: 4.40 ± 0.77 years), half of whom had a family history of reading disorder. The family history group showed slower proportional growth (relative to total brain volume) in the left supramarginal and inferior frontal gyri, and faster proportional growth in the right angular, right fusiform, and bilateral lingual gyri. This suggests delayed development of left hemisphere reading areas in children with a family history of dyslexia, along with faster growth in right homologues. This alternate development pattern may predispose the brain to later reading difficulties and may later manifest as the commonly noted compensatory mechanisms. The results of this study further shows our understanding of structural brain alterations that may form the neurological basis of reading difficulties.