Teaching evaluation at many institutions is insufficient to support, recognize, and reward effective teaching. We developed a long-term intervention to support science, technology, engineering, and mathematics (STEM) department heads in advancing teaching evaluation practices. We describe the intervention and systematically investigate its impact on departmental practices within a research-intensive university. The outcomes varied considerably by department, with four departments achieving extensive teaching evaluation reform and seven departments achieving more limited reform. We used qualitative content analysis of interviews and meetings to investigate department head readiness for change and how it related to the reforms they achieved. All department heads perceived inadequacies in their current evaluation practices, but this dissatisfaction did not reliably predict the changes they pursued. Heads only pursued changes that they perceived to have clear benefits. All heads worried that faculty might resist new practices, but heads who were most successful in facilitating change saw ways to work around resistance. Heads who led the most change questioned their own expertise for reforming teaching evaluation and delegated the work of developing new evaluation practices to knowledgeable colleagues. We discuss emergent hypotheses about factors that support heads in challenging the status quo with more robust and equitable evaluation practices.