Numbers and letters are culturally created symbols that acquire meaning through extensive training, significantly influencing brain function. The distinct hemispheric specialization of cortical regions for these categories has been hypothesized to relate to the co-activated brain networks: the left language regions for letters, and the right intra-parietal sulcus for numbers. However, the potential influence of deafness and sign language on hemispheric specialization for letters and numbers remains unclear. The present study aims to explore this issue by using a FPVS-EEG approach with an oddball paradigm. Deaf and hearing children aged 8-13 were exposed to rapid streams of visual stimuli (6 Hz), with a deviant category introduced periodically (every 5 items; at 1.2 Hz) and eliciting a neural response in the frequency domain if discriminated from the base category. Here, digits are served as base stimuli and letters as oddball stimuli, and vice-versa. Our results suggest disparities in hemispheric lateralization for letters between deaf and hearing children, while neural responses to digits did not significantly vary between the two groups. Both groups exhibited right-lateralized responses to digits, which were stronger compared to responses to letters. Importantly, in deaf children, the neural response to letters was stronger in the right hemisphere, whereas hearing children displayed a bilateral response with a nonsignificant trend toward left lateralization. The important implications of these exploratory results, suggesting an early impact of sensory deprivation and/or sign language on the organization of the brain, are discussed.