Intentional synthesis of research findings is necessary to inform practice, particularly within the research on evaluation (RoE) literature. This study expands upon the work of Coryn et al. (2017) to synthesize the RoE studies pertaining to the domain of evaluation context. Findings from this study demonstrate that organization and program's size,…

Automated Essay Scoring (AES) tools aim to improve the efficiency and consistency of essay scoring by using machine learning algorithms. In the existing research work on this topic, most researchers agree that human-automated score agreement remains the benchmark for assessing the accuracy of machine-generated scores. To measure the performance of…

Automated scoring of student responses to open-ended questions, including short-answer questions, has great potential to scale to a large number of responses. Recent approaches for automated scoring rely on supervised learning, i.e., training classifiers or fine-tuning language models on a small number of responses with human-provided score…

This research investigates the effects of rater experience on performance ratings in language testing using a systematic review of studies published from 1985 to 2017. Based on a comprehensive literature search of 14 databases, we identified sixteen relevant papers. With these we conducted a narrative review to conceptualize a theoretical…

Peer assessment is a promising solution for scaling up the grading of a large number of submissions. The reliability of evaluations is one of the critical issues in peer assessment; several probabilistic models have been proposed for obtaining reliable grades from peers. Peer correction is a similar framework, in which students are instructed to…

Since the George Floyd murder in 2020 and Black Lives Matter campaign, there has been increased interest among evaluators and PIs in Diversity, Equity and Inclusion (DEI) in their projects and evaluations. Our previous research showed a lack of clarity in understanding the distinction between the terms in practice and desire to engage with these…

Evaluations were initially employed to measure the effectiveness of programs because evaluators have a unique ability to influence innovative decision-making. Innovation requires not only the generation of credible evidence, but knowledge brokering for utilization by pertinent stakeholders. Federal evaluations can play an integral role in…

Providing timely feedback is crucial in promoting academic achievement and student success. However, for multifarious reasons (e.g., limited teaching resources), feedback often arrives too late for learners to act on the feedback and improve learning. Thus, automated feedback systems have emerged to tackle educational tasks in various domains,…

Massive Open Online Courses (MOOCs) have become more and more popular recently. These courses have attracted a large number of students world-wide. In a popular course, there may be thousands of students. Such a large number of students in one course makes it infeasible for the instructors to grade all the submissions. Peer assessment is thus an…

Research-practice partnerships have become increasingly popular for promoting evidence use, and this is similar to how evaluators partner with practitioners. Researchers and evaluators working in partnership with practitioners participated in a survey measuring evidence use, research rigor and relevance, and interpersonal factors (e.g.,…

Evaluators have opportunities to advance equity within evaluations, yet little research has examined whether and how evaluators center equity in evaluation practice. This paper explores whether and how evaluators in New England address inequities and advance equity throughout evaluation phases. The study uses a complementarity, sequential mixed…

The purpose of this report is to present our process and results for establishing validity and reliability of an observation tool used to investigate teaching practices that high school mathematics teachers use to engage students. We developed our tool using established practices, such as reviewing literature to develop a framework for instruction…

Academic reviews take on new directions in an ever-increasing fast-paced development. Unit reviewers should be provided different samples, with varied cultural contents also providing nuances, as related to the 'object' of the evaluation, as well as a final overall product, i.e. more of a synthesis, through which one will check the degree of…

The main purpose of the study is to evaluate the qualities of human essay ratings for a large-scale assessment using Rasch measurement theory. Specifically, Many-Facet Rasch Measurement (MFRM) was utilized to examine the rating scale category structure and provide important information about interpretations of ratings in the large-scale…

This study examined the use of a popular Automatic Speech Recognition (ASR), Google Voice Typing (GVT), to automatically assess English as second language pronunciation. It aimed to answer the following question: What is the relationship between GVT-rated scores and human-rated scores? To answer this question, we compared audio recordings of 56…