With the vast array of contextual factors, pedagogical approaches, models of implementation, and purposes of education and training related to online learning, educators, learners, and the general public alike are seeking answers regarding utility and effectiveness of online learning. This article identifies and responds to many of the challenges and issues related to the evaluation of online learning that will likely test evaluators in the years and decades to come. The author provides an early conceptualization of a comprehensive evaluation framework to focus evaluation on the many components of online learning rather than a narrow aspect.