Athena Swan was established in 2005 with the goal of advancing the careers of women in STEMM (Science, Technology, Engineering, Mathematics and Medicine) subjects in higher education. Since then, it has expanded its remit in various ways and has generated some controversy. This paper explains how Athena Swan operates as a policy-scoring scheme, requiring universities to submit plans for policy improvement in order to receive a bronze, silver or gold award. We explain that the implications of rewarding plans for improvement have not been widely understood. We document Athena Swan's changing policies on sex and data collection, which for a period discouraged data collection on sex. We demonstrate the way in which activists were able to capture Athena Swan's policy development process and argue that policy-scoring schemes can lead to groupthink and pose risks to academic freedom. The outsourcing of equalities work to such schemes does not encourage evidence-based prioritisation and discussion of the needs of competing characteristics.