Panahi and Mohebbi review James Dean Brown's 50-years of research in language testing, curriculum development and research statistics with reference to an impressionistic framework for analysis containing two components with their subcomponents: Annotations (i.e., briefing and implications) and main concepts and themes (i.e., testing and teaching…

There is significant controversy over what counts as evidence in the evaluation of social interventions. It is increasingly common to use methodological criteria to rank evidence types in a hierarchy, with Randomised Controlled Trials (RCTs) at or near the highest level. Because of numerous challenges to a hierarchical approach, this article…

This article discusses the nonuse, misuse, and proper use of pilot studies in experimental evaluation research. The authors first show that there is little theoretical, practical, or empirical guidance available to researchers who seek to incorporate pilot studies into experimental evaluation research designs. The authors then discuss how pilot…

Our understanding of programs is enhanced when trained, skilled, and observant evaluators go "into the field"--the real world where programs are conducted--paying attention to what's going on, systematically documenting what they see, and reporting what they learn. The article opens by presenting and illustrating twelve reasons for…

The Council for the Accreditation of Educator Preparation (CAEP) requires faculty from educator preparation programs to provide evidence of credential candidates' impact on K-12 student learning. However, there is a paucity of information on preparation programs' use of direct assessments of student learning to gauge credential candidate…

The important but little understood problem that motivated this study was the lack of research on valid assessment methods to determine progress in higher-order learning in situations involving complex and ill-structured problems. Without a valid assessment method, little progress can occur in instructional design research with regard to designing…

Some researchers have argued for the use of or have attempted to make use of randomization tests in single-subject research. To address this tide of interest, the authors of this article describe randomization tests, discuss the theoretical rationale for applying them to single-subject research, and provide an overview of the methodological…

This article provides two illustrations of some of the factors that can influence findings from pre- and post-test research designs in evaluation studies, including regression to the mean (RTM), maturation, history and test effects. The first illustration involves a re-analysis of data from a study by Marsden (2004), in which pre-test scores are…

Background: Empathy is an individual's capacity to understand the behavior of others, to experience their feelings, and to express that understanding to them. Empathic ability is an asset professionally for individuals, such as teachers, physicians and social workers, who work with people. Being empathetic is also critical to our being able to…

Beginning in 2002, The Chicago Community Trust embarked on an ambitious grant- making strategy to improve education outcomes primarily in the city of Chicago and Cook County, Illinois. Known as The Education Initiative, this effort focused on three priority grant areas: literacy, professional development, and alternative models of schools. In…

Although the general conceptual basis appeared far earlier, theory-driven evaluation came to prominence only a few decades ago with the appearance of Chen's 1990 book "Theory-Driven Evaluations." Since that time, the approach has attracted many supporters as well as detractors. In this paper, 45 cases of theory-driven evaluations, published over a…

The purpose of this paper is to provide a basis for judging the adequacy of evaluation plans or, as they are commonly called, evaluation designs. The authors assume that using the procedures suggested in this paper to determine the adequacy of evaluation designs in advance of actually conducting evaluations will lead to better evaluation designs,…

I define a treatment effect in terms of a comparison of outcomes and provide a typology of all possible comparisons that can be used to estimate treatment effects, including comparisons that are relatively unknown in both the literature and practice. I then assess the relative merit, worth, and value of all possible comparisons based on the…

Good evaluation requires that evaluation efforts themselves be evaluated. Many things can and often do go wrong in evaluation work. Accordingly, it is necessary to check evaluations for problems such as bias, technical error, administrative difficulties, and misuse. Such checks are needed both to improve ongoing evaluation activities and to assess…

Higher education experts tout learning outcomes assessment as a vehicle for program improvement. To this end the authors share a rubric designed explicitly to evaluate the quality of assessment and how it leads to program improvement. The rubric contains six general assessment areas, which are further broken down into 14 elements. Embedded within…