Infants diagnosed with autism spectrum disorder (autism) have difficulty engaging in social communication and interactions with others and often experience language impairment. The use of infant-directed speech (IDS), which is the speech register used when interacting with infants, is associated with infant language and socio-communicative development. The aim of this study was twofold; the first aim was to scope the literature to determine if evidence exists for differences between the IDS caregivers use to infants at high-risk or those later diagnosed with autism, and the IDS typically spoken to neurotypical infants. The second aim was to investigate if any IDS characteristics used by caregivers of high-risk or diagnosed infant populations predicted language development. Twenty-six studies were included and provided evidence that high-risk and later diagnosed infants are exposed to similar amounts of IDS as their neurotypical peers. There is evidence, however, that the IDS used with high-risk and later diagnosed infants may comprise shorter utterances, more action-directing content, fewer questions, more attention bids, and more follow-in commenting. There is also evidence that more attention bids and follow-in commenting used to infants at high risk or those later diagnosed with autism were associated with better language abilities longitudinally.