Individual differences in working memory (WM) influence reading skills. We aim to identify how different domains of WM explain reading performance, and how this association changes with age and reading expertise. Hungarian children from first to sixth grade took part in our study. The decoding skills of all children were assessed with 1-min word reading and pseudoword reading tasks. Furthermore, children participated in one of three WM tasks: a visuospatial, a verbal, and a phonological WM task, yielding three groups. Moderation analyses examined how the relationship between reading and WM changes with increasing school grades. We observed a decreasing role of visuospatial and verbal WM in predicting reading. No age-related change was detected in the prediction of reading skills from phonological WM. Our study suggests that visuospatial and verbal WM are crucial at the beginning of reading, while the role of phonological WM in reading does not change during the elementary school years. Implications for reading instruction are discussed.