In this study, we investigated the impact of the evaluation capacity building (ECB) efforts of an organization by examining the evaluation plans included in funding proposals over a 14-year period. Specifically, we sought to determine the degree to which and how evaluation plans in proposals to one National Science Foundation (NSF) program changed…

Science, Technology, Engineering, and Mathematics (STEM) education has become increasingly important in the US due to its influence on the nation's educational needs, the creation of a skilled labor force, and opportunities for more tech-savvy workers. However, the evaluation approaches and methodologies used in STEM education programs have come…

We propose a new framework for conceptualizing measures of instructed second language (L2) pronunciation performance according to three sets of parameters: (a) the constructs (focused on global vs. specific aspects of pronunciation), (b) the scoring method (human raters vs. acoustic analyses), and (c) the type of knowledge elicited (controlled vs.…

Although much research has compared the functioning between analytic and holistic rating scales, little research has compared the functioning of binary rating scales with other types of rating scales. This quantitative study set out to preliminarily and comparatively validate binary and analytic rating scales intended for use in formative…

The literature on Automated Essay Scoring (AES) systems has provided useful validation frameworks for any assessment that includes AES scoring. Furthermore, evidence for the scoring fidelity of AES systems is accumulating. Yet questions remain when appraising the scoring performance of AES systems. These questions include: (a) which essays are…

We systematically evaluated the peer-reviewed Rorschach validity literature for the 65 main variables in the popular Comprehensive System (CS). Across 53 meta-analyses examining variables against externally assessed criteria (e.g., observer ratings, psychiatric diagnosis), the mean validity was r = 0.27 (k = 770) as compared to r = 0.08 (k = 386)…

In this commentary, we examine Braverman's insights into the trade-offs between feasibility and rigor in evaluation measures and reject his assessment of the trade-off as a zero-sum game. We, argue that feasibility and policy salience are, like reliability and validity, intrinsic to the definition of a good measure. To reduce the tension between…

In schools across America, teachers know who among their peers is doing the best work and who is not. Yet the nation's evaluation systems tend to foster the notion that all teachers perform the same way, with the same results for students. Indeed, in an attempt at equality--uniform treatment for everyone--current evaluation systems often end up…

This "Stated Briefly" report is a companion piece that summarizes the results of another report of the same name. REL Northeast and Islands, in collaboration with the Northeast Educator Effectiveness Research Alliance and the New Hampshire Department of Education conducted a study of the implementation of new teacher evaluation systems…

REL Northeast and Islands, in collaboration with the Northeast Educator Effectiveness Research Alliance and the New Hampshire Department of Education, conducted a study of the implementation of new teacher evaluation systems in New Hampshire's School Improvement Grant (SIG) schools. While the basic system features are similar across district…

Background: Readability tests are indicators that measure how easy a document can be read and understood. Simple, but very often ignored, readability statistics cannot only provide information about the level of difficulty of the readability of particular documents but also can increase an evaluator's credibility. Purpose: The purpose of this…

Methodological rigor consists of a series of elements that, in combination, determine the confidence with which conclusions can be drawn from the evaluation results. These elements include evaluation design, conceptualization of constructs, measurement strategies, time frames, program integrity, and others. The authors examine the factors that…

In this article, the authors examine evaluation process use in a large multilevel, multisite core evaluation. This case highlights the nature of the evaluation processes of the Collaboratives for Excellence in Teacher Preparation (CETP) core evaluation in terms of four categories: (1) social processing; (2) engagement; (3) involvement; and (4)…

Audit trail analysis can provide valuable insights to researchers and evaluators interested in comparing and contrasting designers' expectations of use and students' actual patterns of use of educational technology environments (ETEs). Sequence analysis techniques are particularly effective but have been neglected to some extent because of real…

Over the past twenty years several topics covered in "New Directions for Evaluation" (NDE) are relevant to underrepresented groups, including cultural sensitivity; cultural competence; stakeholder-based, responsive, democratic deliberation; and participatory evaluation theories, methodologies, and practices. Also, the topic of social justice and…