In this column, the author revisits who we are as assessment professionals, how we are trained, what our role is, and what our responsibilities are--to ourselves as a community, to our colleagues in education, at our institution, and to those we serve-- especially from an equity lens. They also share a few strategies to address the gaps they see.

The European Research Council (ERC) receives many high-quality applications, but funds only a few. We analyze how members of ERC review panels assess applications in the first, highly competitive step of evaluations for ERC Starting and Consolidator Grants. Drawing on interviews with ERC panel members in different fields, we show that they adopt a…

In this article, the authors share their reflections on how different experiences and paradigms have broadened their understanding of the work of assessment in higher education. As they collaborated to create a panel for the 2024 International Conference on Assessing Quality in Higher Education, they recognized that they, as assessment…

Background: Rapid reviews involve a streamlined approach to knowledge synthesis. They are used to identify high-quality evidence for the purpose of informing decisions and initiatives, completed over relatively short timeframes, and have been found to reach conclusions that do not differ extensively from full systematic reviews. Although common in…

This essay provides advice for effectively reviewing conference proposals, including how to write comments that are helpful to proposal authors, how to use the "Comments to Program Chair" box, and issues to consider when assigning proposal ratings and recommending acceptance or rejection. Several benefits of reviewing proposals are…

This article presents a new tool called Critical Evaluation Capital (CEC) designed to address issues of equity and social justice in program evaluation. CEC is grounded in the tenants of critical race theory and inspired by Yosso's work on community cultural wealth which raises critical issues of positionality and access. CEC is a system for…

This article is the fifth in a series on the relationship of evaluation theory and practice. It was commissioned by the Eastern Evaluation Research Society and the American Evaluation Association as part of The Chelimsky Forum, launched in 2013 to honor Eleanor Chelimsky, one of the most influential and respected program evaluators of our era, and…

Assessment plays an important role in learning and Multiple Choice Questions (MCQs) are quite popular in large-scale evaluations. Technology-enabled learning necessitates a smart assessment. Therefore, automatic MCQ generation became increasingly popular in the last two decades. Despite a large amount of research effort, system generated MCQs are…

Researchers in many disciplines work with ranking data. This data type is unique in that it is often deterministic in nature (the ranks of items "k"-1 determine the rank of item "k"), and the difference in a pair of rank scores separated by "k" units is equivalent regardless of the actual values of the two ranks in…

In this paper we outline the use of a concept mapping narrative method to elicit the complex and interconnected conceptualisations of specialized knowledge (assessment literacy) for a specific cohort of participants (teacher-examiners). In particular, we are interested in exploring how relational concept mapping and narrative interviews can be…

Recent public events have unnerved the country, touched the consciousness of the populace, and reveal the resurgence of the White supremacist movement, causing many citizens to reflect on race relations. Given the profound impact of racism deeply embedded in the fabric of this country and its complex and destructive influence on the lives of…

Many studies of education interventions make claims about impacts on students' outcomes. Some studies have designs that enable readers to make causal inferences about the effects of an intervention but others have designs that do not permit these types of conclusions. To help policymakers, practitioners, and others make sense of study results, the…

In 2020, because of the COVID-19 pandemic the higher education sector, in the United Kingdom and internationally, transitioned to online assessment, at a speed and scale which might have been unimaginable under normal circumstances. The priority in the sector was to ensure that fundamental principles of assessment, including integrity, were…

Evaluation professionals need to be nimble and innovative in their approaches in order to be relevant and provide useful evidence to decision-makers, stakeholders, and society in the crowded public policy landscape. In this article, we offer serious games as a method that can be employed by evaluators to address three persisting challenges in…

To begin statistical analysis, Bayesians quantify their confidence in modeling hypotheses with priors. A prior describes the probability of a certain modeling hypothesis apart from the data. Bayesians should be able to defend their choice of prior to a skeptical audience. Collaboration between evaluators and stakeholders could make their choices…