This study investigated age differences in the utilization of visually contrastive information (i.e., differently colored identical objects) for temporary referential ambiguity resolution during spoken sentence comprehension. Five- and 6-year-old Japanese children and adults listened to sentences that contained a color adjective-noun combination and identified the object that the noun phrase referred to from a visual scene. We examined the effects of visually contrastive information on either fixations on referent/competitor objects or pupil dilations. The results showed that participants of all age groups directed fewer gazes toward the referent object after they heard the adjectives, when the visually contrastive object was the competitor (i.e., not the referent). The 5-year-olds also directed fewer gazes toward the referent when the contrastive object was the referent. With regard to pupil dilations, no significant effects emerged among adults; however, the pupil dilations of 6-year-olds were larger when the displays contained visually contrastive information than when they did not. The pupil dilations of 5-year-olds were smaller when the referent was one of the contrastive objects. These results suggest that young children's utilization of visually contrastive information is not fully mature and that it may begin with scalar adjectives and eventually generalize to non-scalar adjectives.