Some people in Japanese society have been classified as 'invisible' because they could not complete their compulsory elementary school education and have failed to assimilate into mainstream society. The 2010 Japan Census identified about 1.3 people per thousand over 15 years old who had not graduated from elementary school. However, the circumstances that prevented them from completing their compulsory elementary school education and their current challenges in accessing lifelong learning have yet to be researched. In an attempt to investigate these two issues, quasi-experimental panel data analysis with count data was applied to determine the characteristics of invisible people using the Japan census data disaggregated by the district for all residents and decomposed of environmental and war factors, thus unravelling the geographic anomaly in the national data. Estimation results showed that their features can be explained by three multi-level layers: 1) an explanation due to personal conditions: being in a state of poverty, 2) an explanation created by differences in nationality, and 3) identification from environmental factors, such as geographical isolation and the Battle of Okinawa. We can identify specific features of how to deliver lifelong learning to people with information barriers.