Offering explanations is a central part of teaching mathematics, and understanding those explanations is a vital activity for learners. Given this, it is natural to ask what makes a good mathematical explanation. This question has received surprisingly little attention in the mathematics education literature, perhaps because the field has no agreed method by which explanation quality can be reliably assessed. In this paper, we explore this issue by asking whether mathematicians and undergraduates agree with each other about explanation quality. A corpus of 10 explanations produced by 10 mathematicians was used. Using a comparative judgement method, we analysed 320 paired comparisons from 16 mathematicians and 320 from 32 undergraduate students. We found that both mathematicians and undergraduates were able to reliably assess the quality of a set of mathematical explanations. Furthermore, the assessments were largely consistent across the two groups. Implications for theories of mathematical explanation are discussed. We conclude by arguing that comparative judgement is a promising technique for exploring explanation quality.