This paper presents findings from a collaborative project on critical visual literacy in primary schools. In the project, we (a researcher and a teacher) implemented a series of picture book discussions with children in years 5 and 6. Our first aim was to develop the children's ability to analyse the visual images in picture books and how authors/illustrators use these, together with writing, to communicate specific ideas. The second aim was to promote a dialogic lesson in which the children would lead the talk. In the paper, I discuss the challenges we faced in reaching our aims. My data include observation notes and audio recordings of the sessions and of discussions with the children. In the analysis, I examine the roles we took on, for example, as extenders and clarifiers of the children's talk. However, unbeknown to us at the start, our two project aims turned out to be in conflict with each other. While the children enjoyed looking closely at the pictures and discussing the books' possible meanings, without repeated questions and prompts, they focused on the book as story not on how it was constructed by the author and illustrator. This required a more teacher-focused approach than what we had planned.