Although teachers have access to a great deal of scholarship on teaching academic vocabulary (AV), much less is known about the experiences and perceptions of English language learners regarding AV. To address this gap, we used an online metaphor elicitation survey and follow-up interviews to collect data from 432 undergraduates at a Hong Kong university on their experience of learning AV. For this paper, we focus on the 199 students whose metaphors related to difficulties or challenges. A thematic analysis of this data showed that about two-thirds of this subgroup perceived AV to be inherently difficult, while for the other third, the challenge primarily concerned their negative perceptions of their experience or abilities. We conclude by arguing for the value of understanding learner perspectives and proposing specific teaching strategies for addressing each of the perceived challenges.