This study explores the relationship between engagement with an online, free-to-use question-generation application (PeerWise) and student achievement. Using PeerWise, students can create and answer multiple-choice questions and can provide feedback to the question authors on question quality. This provides further scope for students to engage in discussion about the question with their peers. Data on PeerWise use and examination performance were collected from over 3000 students across six large undergraduate courses (in physics, chemistry and biology) over three academic years in three research-intensive UK universities. A reliable and valid measure of overall PeerWise activity was created and a multilevel model developed describing the relationship between PeerWise activity and student performance in end of course examinations. Using this approach, a significant positive association was found between students' engagement with PeerWise and their academic attainment in end of course exams, even controlling for prior ability. The implications of these findings for educators are discussed.