Numerous studies investigate high-stakes personnel evaluation systems in education, but nearly all focus on evaluation of teachers. We instead examine the evaluation of school principals at scale using data from the first 4 years of implementation of Tennessee's multiple-measure administrator evaluation system. We focus specifically on the rubric-based practice ratings given by principals' supervisors that constitute one half of principals' overall evaluation scores. We find that supervisors' ratings are internally consistent, relatively stable over time, and predictive of other performance measures, such as student achievement growth and teachers' ratings of school leadership quality. However, raters fail to differentiate dimensions of principal practice, and ratings may be biased by factors, such as school poverty, outside the principal's control.