Different speakers may pronounce the same sounds very differently, yet listeners have little difficulty perceiving speech accurately. Recent research suggests that listeners adjust their preexisting phonemic categories to accommodate speakers' pronunciations ("perceptual learning"). In some cases, these adjustments appear to reflect general changes to phonemic categories, rather than speaker-specific adjustments. But what happens when listeners encounter multiple speakers with different pronunciations? We exposed listeners to two speakers who varied in their pronunciation of a particular phoneme (Experiment 1: /d/ or /t/; Experiment 2: /s/ or /esh/). Listeners then categorized sounds on /d/-/t/ or /s/-/esh/ continua, in the same two voices. The results suggest that perceptual experience leads to very different learning for different types of phonemic contrasts. For fricatives, perceptual learning was speaker-specific: The system was able to maintain multiple different representations simultaneously. In contrast, perceptual learning for stop consonants resulted in more general changes that required the system to re-adjust when a new pronunciation was encountered.