Belief in genetic determinism (BGD) has been associated with increased social stereotyping and prejudicial views and consequently is of significant concern to science educators. The Public Understanding and Attitudes towards Genetics and Genomics (PUGGS) instrument was developed to measure relationships among BGD, genetics knowledge, and demographic variables. PUGGS validity evidence has relied primarily on Classical Test Theory frameworks and Brazilian samples. Using a more advanced psychometric framework (Rasch analysis) and a large North American undergraduate sample (n > 800), we further evaluate validity claims by studying (1) dimensionality and function of PUGGS item sets; (2) magnitudes of item endorsement across human traits (social, biological) and taxonomic (animal, plant) categories; and (3) degree of trait-level genetic overattribution. Similar to Gericke et al. (Sci Educ 26:1223-1259, 2017), we identified a two-dimensional structure for the BGD scale (i.e., social, biological) and the genetics knowledge scale (i.e., gene-environment interactions [GEI], genetics and genomics knowledge [GGK]). However, there were several problems with the functioning of the item sets (e.g., low reliability for GEI, problematic rating scale for BGD biological). We report that the magnitudes of GEI and GGK did not differ by taxonomic context. Finally, genetic over- (and under-) attribution was identified for both biological and social traits, indicating that students harbored considerably diverse and frequently non-normative conceptions about genetic contributions to traits. Importantly, psychometric and theoretical concerns reported here raise questions about the operationalization of the PUGGS BGD construct. Recommendations for PUGGS revisions and educational implications are discussed.