The purpose of this research is to develop a vocabulary assessment, Evaluation of Academic Vocabulary or "EAV," that gauges students' depth of knowledge of academic words and is sensitive to vocabulary growth in the context of literacy interventions. We report results from two studies with native English-speaking middle school students designed to inform the development and pilot test the utility and technical qualities of an innovative vocabulary assessment that is sensitive to students' level of understanding of general academic words. Across the two studies, we employed a mixed methods research approach. In the first study, we drew on classic psychometric assessment methods, Bayesian network methods, signal detection theory to evaluate the validity and technical qualities of the assessment. In the latter study, we conducted cognitive interviews to further validate the assessment. Results suggest that: (1) the assessment captures growth in knowledge of general academic words, i.e., words that are important for the comprehension of academic texts; and (2) it is sensitive to treatment effects academic vocabulary intervention for adolescents. [This article was published in "Language Assessment Quarterly" v16 n2 p196-216 2019 (EJ1222377).]