An assumption underlying speaking tests is that scores reflect the ability to produce online, non-rehearsed speech. Speech produced in testing situations may, however, be less spontaneous if extensive test preparation takes place, resulting in memorized or rehearsed responses. If raters detect these patterns, they may conceptualize speech as inauthentic. As of yet, no studies have investigated raters' perceptions or viewpoints on authenticity. In this exploratory study, 58 raters rated eight speech samples, one set of four recorded by test takers who had been exposed to a test prompt one week beforehand and a second set of four who had not been exposed. The 58 raters rated the samples on 5 continuous speech-production authenticity indicators and 4 continuous proficiency indicators. Seven raters additionally participated in a stimulated verbal recall. The raters were able to differentiate authenticity across the exposure sets. Raters with experience working in China (n = 42), an educational context prone to cram-test preparation practices, were even better able to do so. The stimulated recall revealed a range of criteria raters used in their judgements of authenticity. In this study I discuss these findings and how this hidden facet may play a role in spoken-performance-test rating.