Children living in low-resource settings are at risk for failing to reach their developmental potential. While the behavioral outcomes of growing up in such settings are well-known, the neural mechanisms underpinning poor outcomes have not been well elucidated, particularly in the context of low- and middle-income countries. In this study, we measure brain metabolic responses to social and nonsocial stimuli in a cohort of 6- and 36-month-old Bangladeshi children. Study participants in both cohorts lived in an urban slum and were exposed to a broad range of adversity early in life including extreme poverty, malnutrition, recurrent infections, and low maternal education. We observed brain regions that responded selectively to social stimuli in both ages indicating that these specialized brain responses are online from an early age. We additionally show that the magnitude of the socially selective response is related to maternal education, maternal stress, and the caregiving environment. Ultimately our results suggest that a variety of psychosocial hazards have a measurable relationship with the developing social brain.