Many states report school performance grades as a way to inform the public about school quality. However, past research has shown that when these grades drew largely on proficiency-based measures, they served to capture variation in school and community demographics rather than school quality. We extend this literature by examining whether a multidimensional measure of school quality such as those required under the Every Student Succeeds Act is less confounded by out-of-school factors than the proficiency measures that characterized previous generations of accountability. Drawing on school accountability grades from Florida combined with school and community demographic data, we find that more than half the variation in multidimensional measures of school quality can be explained by observable school- and county-level factors outside the school's locus of control. Together, our findings show that even school grades that draw on multiple measures misattribute the contribution of demographics and socioeconomics to school quality--but subcomponents based on learning gains perform better than those based on proficiency. We conclude with policy implications and recommend that states focus public reporting on school quality measures driven less by out-of-school factors and more by the school's true contribution to student outcomes.