Understanding which factors can impact rater judgments in assessments is important to ensure quality ratings. One such factor is whether prior performance information (PPI) about learners influences subsequent decision making. The information can be acquired directly, when the rater sees the same learner, or different learners over multiple…

This method note presents Q methodology as a useful tool for evaluators to add to their practice toolbox. Q methodology, which involves both quantitative and qualitative techniques, can help researchers and evaluators systematically understand subjectivity and the communicability of opinions and perspectives. We first provide an overview of Q…

Intentional synthesis of research findings is necessary to inform practice, particularly within the research on evaluation (RoE) literature. This study expands upon the work of Coryn et al. (2017) to synthesize the RoE studies pertaining to the domain of evaluation context. Findings from this study demonstrate that organization and program's size,…

Institutions of higher education are asked to consider how their work can advance equity in institutional outcomes. Assessment, too, has been asked to consider the ways in which traditional student assessment "privileges and validates certain types of learning and evidence of learning over others" (Montenegro and Jankowski 2017, p. 5).…

In the context of policy-driven language testing for citizenship, a growing body of research examines the political justifications and ethical implications of language requirements and test use. However, virtually no studies have looked at the role that language testers play in the evolution of language requirements. Critical gaps remain in our…

Classroom observational protocols, in which raters observe and score the quality of teachers' instructional practices, are often used to evaluate teachers for consequential purposes despite evidence that scores from such protocols are frequently driven by factors, such as rater and temporal effects, that have little to do with teacher quality. In…

Unfolding models are frequently used to develop scales for measuring attitudes. Recently, unfolding models have been applied to examine rater severity and accuracy within the context of rater-mediated assessments. One of the problems in applying unfolding models to rater-mediated assessments is that the substantive interpretations of the latent…

This article explores the utility of mindfulness in the field of evaluation. Mindfulness is a translation of the ancient Indian word, "Sati," which means awareness, attention, and remembering. While definitions vary, a practical definition of mindfulness is present-moment awareness in an open and nonjudgmental manner. Mindfulness-based…

The teacher evaluation process has been an area of study for decades because of its complexity. Going back to one-room schoolhouses, teaching has always been an isolated profession--usually a room, a teacher, and students. Other colleagues or administrators are not participants in the classroom most of the time, which creates difficulties for…

Due to subjectivity in oral assessment, much concentration has been put on obtaining a satisfactory measure of consistency among raters. However, the process for obtaining more consistency might not result in valid decisions. One matter that is at the core of both reliability and validity in oral assessment is rater training. Recently,…

It is widely acknowledged in research that common criteria and aligned standards do not result in consistent assessment of such a complex performance as the final undergraduate thesis. Assessment is determined by examiners' understanding of rubrics and their views on thesis quality. There is still a gap in the research literature about how…

This study aimed to measure the effect of rater training given to improve the peer assessment skills of secondary school students on rater behaviors using the many-facet Rasch Measurement model. The research employed a single-group pretest-posttest design. Since all raters scored all students, the analyses were carried out in a fully crossed (s x…

Districts spend thousands of dollars on computerized teacher screeners without knowing if they are identifying the most effective teacher. Hiring quality staff is one of the most important job functions of a principal, and many times a teacher screener score may eliminate an effective teacher. The current study examined the value of teacher…

This study's purpose is to extend our understanding of school leadership for student learning by identifying high-leverage teacher evaluation practices that improve teaching. A partnership with a state education agency administered a teacher questionnaire regarding evaluation practices multiple times in one semester, then linked teacher responses…

We have limited understanding of why reviewers tend to strongly disagree when scoring the same research proposal. Thus far, research that explored disagreement has focused on the characteristics of the proposal or the applicants, while ignoring the characteristics of the reviewers themselves. This article aims to address this gap by exploring…