Measuring the Academic Skills of University Students is a procedure developed in the 1990s at the University of Sydney's Language Centre to identify students in need of academic writing development by assessing examples of their written work against five criteria. This paper reviews the literature relating to the development of the procedure with a focus on studies exploring its reliability and validity. It then describes a study in which teams of language experts and subject experts used the procedure to rate assignments produced by students taking three distance-learning courses. This study provides further insight into the psychometric properties of the procedure: the overall ratings had satisfactory internal consistency and test-retest reliability and were highly correlated with the marks awarded to the assignments analyzed. However, the ratings were also highly correlated with each other and yielded just one principal component, suggesting that even skilled assessors were unable to differentiate between different aspects of academic writing. We therefore conclude that the procedure is a reliable and valid means of identifying students who need writing skills development but that it should not be relied on to identify their particular areas of weakness. (Contains 5 tables.)