Since 2007, two taxonomies have been proposed to identify the components of evaluation practice that may be specified in an evaluation policy. Little is known, however, about how these taxonomies align with evaluation policies developed by philanthropic foundations. Through thematic analysis, this article first compares 12 foundation evaluation…

The reason for conducting the present case study was the problems the researchers encountered during data collection for another research project (Primary Study) entitled "The effects of virtual versus traditional flipped classes on EFL learners' grammar knowledge, self-regulation, and autonomy." Two online questionnaires were…

Assessment rubrics are being promoted and introduced into tertiary teaching practices on the grounds that they are an efficient and reliable tool to evaluate student performance effectively and promote student learning. However, there has been little discussion on the value of using assessment rubrics in higher education. Rather, they are being…

The purpose of this statement is to inform those using or considering the use of value-added models (VAM) about their scientific and technical limitations in the evaluation of educators and programs that prepare teachers. The statement briefly reviews the background and current context of using VAM for evaluations, enumerates specific psychometric…

Cost-effectiveness analysis is rarely used in education. When it is used, it often fails to meet methodological standards, especially with regard to cost measurement. Although there are occasional criticisms of these failings, we believe that it is useful to provide a listing of the more common concerns and how they might be addressed. Based upon…

This report describes the implementation of professional development (PD) reforms and efforts to use teacher effectiveness (TE) data to inform PD through the third year of the initiative for all seven sites: Hillsborough County Public Schools (HCPS), Shelby County Schools (SCS, formerly Memphis City Schools), Pittsburgh Public Schools (PPS), and…

This paper examines the construct measured by automated essay scoring (AES) systems. AES systems measure features of the text structure, linguistic structure, and conventional print form of essays; as such, the systems primarily measure text production skills. In the current state-of-the-art, AES provide little direct evidence about such matters…

Despite considerable interest in the topic of instructional quality in research as well as practice, little is known about the quality of its assessment. Using generalizability analysis as well as content analysis, the present study investigates how reliably and validly instructional quality is measured by observer ratings. Twelve trained raters…

In their article Ruscio et al. (Ruscio, Seaman, D'Oriano, Stremlo, & Mahalchik, this issue) present a comparative study of some of the different variants of the "h" index. The study evaluates a total of 22 metrics, including the "h" index and "h"-type indices, as well as other conventional measures. The novelty of their work is to a large extent…

When the researcher proposed this study, no instrument existed for measuring citizens' attitudes toward school funding equity, so this study was designed as a series of investigations leading to the creation of such an instrument. In order to accomplish this purpose, the researcher first generated an initial pool of items to measure attitudes…

Ruscio and colleagues (Ruscio, Seaman, D'Oriano, Stremlo, & Mahalchik, this issue) provide a thoughtful empirical analysis of 22 different measures of individual scholarly impact. The simplest metric is number of publications, which Simonton (1997) found to be a reasonable predictor of career trajectories. Although the assessment of the scholarly…

Ruscio et al. (Ruscio, Seaman, D'Oriano, Stremlo, & Mahalchik, this issue) write of a thing with which scientists and scholars are all too familiar, the assessment of published research and of its authors. The author was startled to discover how little the agenda of the paper seems to engage with factors one relies on for salary and promotion…

This experimental project investigated the reliability and validity of rubrics in assessment of students' written responses to a social science "writing prompt". The participants were asked to grade one of the two samples of writing assuming it was written by a graduate student. In fact both samples were prepared by the authors. The…

Lecoutre, Lecoutre, and Poitevineau (2010) have provided sophisticated grounding for "p[subscript rep]." Computing it precisely appears, fortunately, no more difficult than doing so approximately. Their analysis will help move predictive inference into the mainstream. Iverson, Wagenmakers, and Lee (2010) have also validated…

Ruscio, Seaman, D'Oriano, Stremlo, and Mahalchik (this issue) evaluate 22 bibliometric indicators, including conventional measures, like the number of publications, the "h" index, and many "h" index variants. To assess the quality of the indicators, their well-justified criteria encompass conceptual, empirical, and practical…