While California Common Core State Standards in Mathematics (CA CCSSM) call for rigorous mathematics for all students, students with disabilities have not been provided equal access to instruction that meets these standards. Universal Design for Learning (UDL) is a research-based framework to develop strategic, expert learners within classroom settings that maximize engagement of a wide variety of students, including students with disabilities. A critical part of the UDL process is the identification of barriers that impede learning. To explore these barriers, the author draws on narratives from memoirs and interviews written by people with learning disabilities about their experiences learning mathematics. These barriers include: (1) Limited avenues for learning mathematics in traditional instruction; (2) Focus on speed and memorization; (3) Limited connections to concepts; and (4) Emotional aspects of mathematics. This brief describes how UDL can provide the foundation of mathematics instruction that provides access to rigorous, standards-based mathematics for all students in California.