The present paper questions the content validity of the eight creativity-related self-report scales available in PISA 2022's context questionnaire and provides a set of considerations for researchers interested in using these indexes. Specifically, we point out some threats to the content validity of these scales (e.g., "creative thinking self-efficacy, creativity, and openness to intellect") including ambiguity in construct and content definition, construct underrepresentation, or item overlaps between constructs. We then discuss content validity from a theoretical angle, outlining how important is the clarity of content domain definition and corresponding test content, to avoid possible jingle-jangle fallacies in the item sets (e.g., overlap between same/similar items attributed to different constructs) which undermine the utility of resulting scores. We also present approaches for empirically quantifying content validity. In all, this paper calls for a cautious use and interpretation of creativity self-report data in PISA 2022 as it stands. It provides guidance for further examination of the content validity of the creativity-related self-report scales in PISA 2022. Such future efforts are necessary to enhance the validity of inferences made from these self-report scale scores, to optimize their comparability across different research teams and nations, toward a more generalizable use in creativity and education research.