The New General Service List Test (NGSLT) (Stoeckel & Bennett, 2015) was designed as a diagnostic test to measure students' written receptive vocabulary knowledge. This test battery was developed based upon the New General Service List (NGSL) (Browne, 2013), which makes it appealing to teachers in Japan, and especially those who see vocabulary as key to English as a foreign or second language learning. The research focused on finding out whether and to what degree the test accurately and reliably measures students' vocabulary knowledge, and to find if there are any incongruences with the scores on this test and those on extraneous standards. Three versions of the NGSLT were distributed and a triangulation method was used to analyze the data, with the findings suggesting that the NGSLT may be less a measure of students' knowledge of the target words than a measure of how well they can understand the answer choices. [For the complete volume of short papers, see ED590612.]