Contending with talker variability has been found to lead to processing costs but also benefits by focusing learners on invariant properties of the signal, indicating that talker variability acts as a desirable difficulty. That is, talker variability may lead to initial costs followed by long-term benefits for retention and generalization. Adult participants learned an artificial grammar affording learning of multiple components in two experiments varying in difficulty. They learned from one, two, or eight talkers and were tested at three time points. The eight-talker condition did not impact learning. The two-talker condition negatively impacted some aspects of learning, but only under more difficult conditions. Generalization of the grammatical dependency was difficult. Thus, we discovered that high and limited talker variability can differentially impact artificial grammar learning. However, talker variability did not act as a desirable difficulty in the current paradigm as the few evidenced costs were not related to long-term benefits.