Purpose: The goal of the current study was to conduct a substantive validity review of four autism knowledge assessments with prior psychometric support (Gillespie-Lynch in J Autism and Dev Disord 45(8):2553-2566, 2015; Harrison in J Autism and Dev Disord 47(10):3281-3295, 2017; McClain in J Autism and Dev Disord 50(3):998-1006, 2020; McMahon in Res Autism Spectr Disord 71:101499, 2020). 69 autism experts who served on the editorial board of one or more peer-reviewed autism journals evaluated the accuracy and ambiguity of autism knowledge questions. 34% of the questions were flagged as "potentially problematic" for accuracy, and 17% of the questions were flagged as "potentially problematic" for ambiguity. Autism expert feedback revealed three themes across ambiguous questions: (1) an oversimplification of mixed or still-evolving research literature, (2) an insufficient recognition of the heterogeneity of the autism spectrum, and (3) a lack of clarity in the question/answer prompt. Substantive validity of future autism knowledge assessments should be carefully evaluated via feedback from a diverse group of autism experts and/or potential respondents. Potentially problematic questions can be removed or modified to improve the validity of autism knowledge assessments.