Much of the existing research on gender wage gaps in China is focused on a singular perspective, such as education, sector, industry, and urban-rural difference. This paper, however, discusses both returns to education and gender wage gaps from a sector perspective. Using Urban Household Survey of 2004 and 2013, this paper presents that the classic human capital theory cannot fully explain the gender wage gap and provides empirical research that multiple labor market segmentations in China exist based on segmentation theory. To investigate return difference among different education stages, we classify the Chinese education system into five categories. First, Heckman sample selection models are used to estimate male and female wages in two sectors. The study then applies quantile regressions to analyze different returns to education across wage groups. Finally, Blinder-Oaxaca decomposition is used to analyze the cause of wage gap by identifying characteristic effect and coefficient effect. Findings indicate that the gender wage gap in the private sector has shrunk but is still greater than that of the public sector; the returns to education are mostly positive for women and higher than that of men; women in the private sector with an educational background of vocational college and below have particular low returns to education; the public sector has a lower characteristic effect because of egalitarianism. The paper concludes by proposing matched policy suggestions centered around guaranteeing enrollment rate, upgrading skills, and protecting labor rights for women, especially those with low education and in the private sector.