Listening to music activates representations of movement and social agents. Why? We test whether causal reasoning plays a role, and find that from childhood, people can intuitively reason about how musical sounds were generated, inferring the events and agents that caused the sounds. In Experiment 1 (N = 120, pre-registered), 6-year-old children and adults inferred the presence of an unobserved animate agent from hearing musical sounds, by integrating information from the sounds' timing with knowledge of the visual context. Thus, children inferred that an agent was present when the sounds would require self-propelled movement to produce, given the current visual context (e.g., unevenly-timed notes, from evenly-spaced xylophone bars). Consistent with Bayesian causal inference, this reasoning was flexible, allowing people to make inferences not only about unobserved agents, but also the structure of the visual environment in which sounds were produced (in Experiment 2, N = 114). Across experiments, we found evidence of developmental change: Younger children ages 4-5 years failed to integrate auditory and visual information, focusing solely on auditory features (Experiment 1) and failing to connect sounds to visual contexts that produced them (Experiment 2). Our findings support a developmental account in which before age 6, children's reasoning about the causes of musical sounds is limited by failure to integrate information from multiple modalities when engaging in causal reasoning. By age 6, children and adults integrate auditory information with other knowledge to reason about how musical sounds were generated, and thereby link musical sounds with the agents, contexts, and events that caused them.