This study documents the development of an assessment (a multiple-choice test with free text rationale) built into an academic skills module over a 2 year period. Initially introduced as a credit bearing assessment to promote student engagement, disappointing results in the first year prompted a further intervention in year 2 with an additional lecture and a series of online resources provided. The results of the assessment from both years were examined; firstly, to determine the success of the intervention, and secondly, the free text rationales in an attempt identify aspects of academic practice that proved difficult for students. With a diverse cohort, the opportunity to examine distinctions between home and international students was also explored. The results indicated that upon the introduction of the intervention the pass rate for the assessment increased from 56% to 80%, however a qualitative analysis of the free text student rationales highlighted persistent underlying problems.