Some of the key outcome measures used in the first federal Child and Family Service Reviews rely on retrospective cohorts and exclude key portions of the population from the analysis. Most discussions of this issue have focused on the extent to which retrospective measurement is a valid basis on which to judge states' performance (Courtney, Needell, & Wulczyn, 2003). The analyses presented here suggest that in some instances the relative or comparative results of retrospective and prospective measurements exhibit few differences. On the other hand, the analyses also make clear that retrospective measurements have two serious deficiencies in relation to improving performance. First, they are likely to identify the wrong issues, and, second, even when they identify the correct issues they fail to provide information needed to improve performance. This article suggests some practical ways in which the information currently available to child welfare agencies can be used to correct these problems. (Contains 5 tables and 3 figures.)