The development of cognitive skills is influenced by characteristics of both individuals and the environments in which they live, including their homes, neighborhoods, schools and workplaces. Contextual features of these environments shape and constrain cognitive development in various ways. This article considers the case of numeracy development within the highly constrained environment of prisons in the United States. Data are analyzed from recent PIAAC assessments of the incarcerated and household populations of the United States. Individuals' assessed numeracy skills and engagement in everyday numeracy practices, both inside and outside of prisons, are analyzed in relation to their education and other background characteristics. Although some of the observed differences in numeracy between household and prison populations can be attributed to differences in these background characteristics, significant differences in numeracy skills and practices remain after controlling for background characteristics. Such regression-adjusted differences may reflect the distinctive numerate environment of prisons. Prisoners' numeracy appears to be initially disrupted by incarceration but gradually improves as it adjusts over time to the new numerate environment. Numeracy also appears to be an important determinant of key social outcomes in prisons. Some methodological limitations of these findings are considered along with their implications for fostering numeracy development and improved social outcomes inside and outside of prisons as well as in other vulnerable populations.