The Fundamentals in Meteorology Inventory (FMI) concept inventory exam was recently developed to gauge understanding of fundamental meteorological concepts among introductory meteorology students. The exam was iteratively tested and refined over several years using a small sample of students. With a goal of evaluating the reliability and validity of the exam for broader instructor use, the FMI was administered to 282 students both pre- and postinstruction at 8 universities across the United States in fall 2017 and fall 2018. A variety of measures of validity and reliability for individual test items as well as whole-test metrics are presented. Overall, the exam appears to be sufficient for measuring the extent of understanding of basic meteorological concepts. This paves the way for future work by instructors and researchers to use the FMI exam as a method to identify levels of and gaps in fundamental meteorological understanding, thus providing opportunities to improve instruction of fundamental concepts.