On the basis of a multistage exploration of evaluation texts, electronic searches, and nominations from the field and from managing social science, the author concludes there is little research literature on managing evaluation studies, evaluators and other workers, and evaluation units. The discussion explores what this limited literature tells…

Research and development of a new type of scoring rubric for the integrated speaking tasks of "TOEFL iBT"® are described. These "analytic rating guides" could be helpful if tasks modeled after those in TOEFL iBT were used for formative assessment, a purpose which is different from TOEFL iBT's primary use for admission…

While much of the work in language testing is concerned with constructing quality tests in order to measure language knowledge in reliable and valid ways, there has been a significant movement in language testing research that examines tests in the context of their use in education and society. This line of research exits from the notion that…

Speaking performance tests typically employ raters to produce scores; accordingly, variability in raters' scoring decisions has important consequences for test reliability and validity. One such source of variability is the rater's level of expertise in scoring. Therefore, it is important to understand how raters' performance is influenced by…

In 1997, American Evaluation Association member George Grob, now retired from the U.S. Department of Health and Human Services (HHS) and currently President of the Center for Public Program Evaluation, made a testimony on Medicare home health care fraud and abuse before the U.S. Senate Special Committee on Aging. The occasion was to announce the…

This article describes an evaluation that was judged to be unsuccessful from the point of view of key program stakeholders. This was due to the fact that the evaluation did not support program advocates who had much to gain from positive evaluation findings. We argue that, although the knowledge needs of stakeholders must be taken into account,…

The full service extended schools (FSES) initiative in England sought to build the capacity of schools to deploy multiple interventions across a range of child, family, and community contexts in order to improve the lives of children and families, and to enhance the sustainability of the areas where they live. As such, it was part of wider…

A simulation study was conducted in an attempt to examine how evaluators modify their evaluation design in response to differing stakeholder groups. In this study, evaluators were provided with a fictitious description of a school-based program. They were then asked to design an evaluation of the program. After the evaluation design decisions were…

Across the nation, states and districts are in the process of building better teacher evaluation systems that not only identify highly effective teachers but also systematically provide data and feedback that can be used to improve teacher practice. "A Practical Guide to Designing Comprehensive Teacher Evaluation Systems" is a tool…

Two case studies are presented to compare and contrast the challenges encountered when attempting to conduct participatory evaluations (P-Es) with tribal programs that represented two extremes of collaboration between the programs and evaluators. In one case, the P-E was successful because the principals were invested in the program, whereas in…

Objective: Multiple trials have found telephone-administered cognitive behavioral therapy (T-CBT) to be effective for the treatment of depression. The aim of this study was to evaluate T-CBT for the treatment of depression among veterans served by community-based outpatient clinics (CBOCs) outside of major urban areas. Method: Eighty-five veterans…

An examination of the scores given using analytic rating scales can provide insight into how well the scales are measuring what they are supposed to be measuring. Many analytic rating scales include separate scales for lexis and grammar, yet the distinction between lexis and grammar has not been widely researched or defined. It is therefore…

This article details an investigation into the onscreen marking (OSM) of Liberal Studies (LS) in Hong Kong--where paper-based marking (PBM) of public examinations is being phased out and wholly superseded by OSM. The study involved 14 markers who had previously rated Liberal Studies scripts on screen in the 2009 Hong Kong Advanced Level…

The oral examination is an important component of the high-stakes "O" Level examination in Singapore taken by 16-17 year olds whose first language may or may not be English. In spite of this, there has been sparse research into the examination. This paper reports findings of an exploratory study which attempted to determine whether there…

This article describes the journey taken by a group of adolescents into the field and practice of youth-led research. The article gives voice to the growing number of youth participating in research and evaluation. The authors give authentic youth accounts of: (1) the process of becoming researchers and evaluators, (2) the benefits and challenges…