The present article considers a fundamental question in evaluation research: "By how much do program effects vary across sites?" The article first presents a theoretical model of cross-site impact variation and a related estimation model with a random treatment coefficient and fixed site-specific intercepts. This approach eliminates…

Evaluation studies can provide feedback on implementation, support continuous improvement, and increase understanding of evaluation systems' impact on teaching and learning. Despite the importance of educator evaluation studies, states often need support to prioritize and fund them. Successful studies require expertise, time, and a shared…

Policymakers are recognizing that Pennsylvania has fallen behind in providing equitable access to high quality early childhood education. Governor Tom Wolf ran on a campaign promise of universal pre-k access and proposed an unprecedented budget increase for early childhood programs in 2016. In Philadelphia, Mayor Jim Kenney has made pre-k a…

The Council for the Accreditation of Educator Preparation (CAEP) requires faculty from educator preparation programs to provide evidence of credential candidates' impact on K-12 student learning. However, there is a paucity of information on preparation programs' use of direct assessments of student learning to gauge credential candidate…

College completion rates in the U.S. are stubbornly low despite the large and rising returns to a college degree. Efforts to increase student success in college have largely ignored a potentially key factor: the instruction that students receive in the sequence of courses that add up to a college education. Improving the quality of instruction may…

In 2010, HCM Strategists received support from the Bill & Melinda Gates Foundation to engage academic researchers in identifying and discussing the best currently available ways to account for differences in college "inputs"--students, resources, and fixed characteristics--when evaluating postsecondary outcomes. The project, Context…

After decades of disinterest, evaluation of the performance of elementary and secondary teachers in the United States has become an important educational policy issue. As U.S. states and districts have tried to upgrade their evaluation processes, one of the models that has been increasingly used is the Framework for Teaching. This paper summarizes…

In this article, a group of inservice providers and beginning researchers describe their experiences in learning to conduct evaluation research on a long-term school-university partnership program. We offer the practical lessons we learned in how to undertake such a study, and we share the immediate and powerful effects that the process of…

The quality and types of relationships formed between students and teachers has been shown to play an essential part in the personal and academic success of students (Davis, 2003; Pianta, 1999). Little, however, has been done to determine the role that assessment plays in teacher-student relationships. Drawing upon the work of cultural…

This research evaluated the inter-rater reliability of the Performance Assessment for California Teachers (PACT). Multiple methods for estimating overall rater consistency include percent agreement and Cohen's Kappa (1960), which yielded discrepancies between rater agreement in terms of whether candidates passed or failed particular PACT rubrics.…

This study has tackled the thorny problem of closing the Student Achievement Gap (SAG) in California's elementary schools. To address that problem, an "Integrated" form of educational leadership called Transformational Instructional Leadership (TIL), a form grounded in "best practices" of Transformational and Instructional…

Described here is a study on the efficacy of a digital media-rich curriculum based on the idea that children can learn best from "media synergy", that is, when children have opportunities to learn skills by engaging in repeated practice with them in many different formats and media (Neuman, 1995). The study is part of the "Ready to…

In order to assess how completely a Small Learning Community (SLC) and/or Career Academy is implemented in any given site, and to connect the degree of implementation with the amount of improvement in student performance, two kinds of information are needed. The first is information on the quality of implementation, which can be collected by…

This study examined the impact of rater agreement on decisions concerning the alignment between the "Golden State Examination in High School Mathematics" (California Department of Education, 2001a) and the University of California (UC) "Statement on Competencies in Mathematics Expected of Entering College Students" (Academic…

Like Dorothy in the Wizard of Oz, faculty members and student affairs educators often believe that the answers they seek about assessment are secrets kept by an all-knowing, powerful person. If they can just find the Wizard of Assessment and be granted the secret assessment formula, all will be well. The student affairs educators at California…