We investigate whether commonly used value-added estimation strategies produce accurate estimates of teacher effects under a variety of scenarios. We estimate teacher effects in simulated student achievement data sets that mimic plausible types of student grouping and teacher assignment scenarios. We find that no one method accurately captures…

From a utilization-focused evaluation perspective, the success of an evaluation is rooted in the extent to which the evaluation was used by stakeholders. This paper details the "Expectations to Change" (E2C) process, an interactive, workshop-based method designed to engage primary users with their evaluation findings as a means of…

Model-based attempts to rigorously study the broad and imprecise concept of "discriminating power" are scarce, and generally limited to nonlinear models for binary responses. This paper proposes a comprehensive framework for assessing the discriminating power of item and test scores which are analyzed or obtained using Spearman's…

Policymakers are recognizing that Pennsylvania has fallen behind in providing equitable access to high quality early childhood education. Governor Tom Wolf ran on a campaign promise of universal pre-k access and proposed an unprecedented budget increase for early childhood programs in 2016. In Philadelphia, Mayor Jim Kenney has made pre-k a…

This chapter presents an introduction to design-based implementation research (DBIR). We describe the need for DBIR as a research approach that challenges educational researchers and practitioners to transcend traditional research/practice barriers to facilitate the design of educational interventions that are effective, sustainable, and scalable.…

In "The Case for Participatory Evaluation in an Era of Accountability" (this issue), Jill Chouinard highlights seven tensions between the participatory and technocratic approaches to evaluation. She suggests that there is a need to create opportunities for more participatory approaches to evaluation. In this essay, I suggest that the era of…

The purpose of this study is to help ensure that strategies for differential item functioning (DIF) detection for students with disabilities are appropriate and lead to meaningful results. We surveyed existing DIF studies for students with disabilities and describe them in terms of study design, statistical approach, sample characteristics, and…

A new approach to identify item clusters fitting the Rasch model is described and evaluated using simulated and real data. The proposed method is based on hierarchical cluster analysis and constructs clusters of items that show a good fit to the Rasch model. It thus gives an estimate of the number of independent scales satisfying the postulates of…

Evaluators have written about the need to consider context in conducting evaluations, but most such admonitions are broad. Context is not developed fully. This chapter reviews the evaluation literature on context and discusses the two areas in which context has been more carefully considered by evaluators: the culture of program participants when…

In 1980 the first meeting of the UK-based Language Testing Forum was held in Lancaster. In 2010, 30 years later, the annual meeting celebrated that first meeting, basing the discussions, as in 1980, on three seminal papers: on the structure of language ability, on communicative language testing, and on testing Languages for Specific Purposes.…

This article raises questions about the use of needs assessment by those seeking to improve the performance of organizations and individuals. While the discrepancy model of needs assessment has enjoyed a long history and is widely accepted, the author states that it has several undesirable consequences in terms of its impact. This article offers…

This article examines the scope for IT-enabled assessments to serve simultaneously both learners and the enterprise of education. The article proposes ways of combining frameworks that come from two different perspectives: 1) a conceptual approach to assessment design for computerized assessment based on evidence-centred design (ECD) and 2) a…

Design-based implementation research offers the opportunity to rethink the relationships between intervention, research, and situation to better attune research and evaluation to the program development process. Using a heuristic called the intervention development curve, I describe the rough trajectory that programs typically follow as they…

Student feedback, test-score growth calculations, and observations of practice appear to pick up different but complementary information that, combined, can provide a balanced and accurate picture of teacher performance, according to research recently released from the Bill & Melinda Gates Foundation. A composite measure on teacher effectiveness…

The current paper on writing assessment surveys the literature on the reliability and validity of essay tests. The paper aims to examine the two concepts in relationship with essay testing as well as to provide a snapshot of the current understandings of the reliability and validity of essay tests as drawn in recent research studies. Bearing in…