Peer review is integral to the evaluation of grant proposals. Reviewer perceptions and characteristics have received limited study, especially their associations with reviewers' evaluations. This mixed methods study analyzed the unstructured comments of 270 experienced peer reviewers after they scored proposals based on mock overall evaluations…

Over the last decade, the idea of societal impact resulting from publicly funded research has changed from being a relatively fringe concern related to high-technology entrepreneurship and spin-off companies to becoming an increasingly important public policy concern. This has stimulated academic science policy research to investigate the impact…

Debates about appropriate, fair and effective ways of assessing research and researchers have raged through the scientific community for decades, recently mostly concerned with discussing the merits and limitations of metric-based, quantitative assessments versus peer review-based, qualitative alternatives. Ample attention has been paid to formal…

This article presents the insights from an evaluation of a transformative policy experiment, in the food domain, conducted at the Swedish Agency of Innovation (Vinnova). To be consistent with the principles and objectives of these policies, it was necessary to implement a formative evaluation approach developed in the Transformative Innovation…

As part of the shift towards a more equitable research culture, funders are reconsidering traditional approaches to peer review. In doing so, they seek to minimize bias towards certain research ideas and researcher profiles, to ensure greater inclusion of disadvantaged groups, to improve review quality, to reduce burden, and to enable more…

We have limited understanding of why reviewers tend to strongly disagree when scoring the same research proposal. Thus far, research that explored disagreement has focused on the characteristics of the proposal or the applicants, while ignoring the characteristics of the reviewers themselves. This article aims to address this gap by exploring…

The European Research Council (ERC) receives many high-quality applications, but funds only a few. We analyze how members of ERC review panels assess applications in the first, highly competitive step of evaluations for ERC Starting and Consolidator Grants. Drawing on interviews with ERC panel members in different fields, we show that they adopt a…

Recent times have seen the growth in the number and scope of interacting professional reform movements in science, centered on themes such as open research, research integrity, responsible research assessment, and responsible metrics. The responsible metrics movement identifies the growing influence of quantitative performance indicators as a…

Journals that adopt open peer review (OPR), where review reports of published articles are publicly available, provide an opportunity to study both review content characteristics and quantitative aspects of the overall review process. This study investigates two areas relevant to the quality assessment of manuscript reviews. First, do journal…

Innovation agencies, that is organizations with the primary focus of funding applied research and technological development, evaluate project proposals to select the most promising proposals for funding. At the moment, there is only little verified knowledge available on project evaluation and selection processes of innovation agencies. We want to…

Evaluation of interdisciplinary, team science research initiatives is an evolving and challenging field. This descriptive, longitudinal, mixed methods case study examined how an embedded, formative evaluation approach contributed to team science in the interdisciplinary Research into Policy to Enhance Physical Activity (REPOPA) project, which…

Evaluators play a central role in assessments of researchers' performance for reward, but the nature of their role and influence is not well understood. Ongoing reliance on evaluator judgement is typically justified as a need for referees in contests for reward, because quantitative performance measures alone can be subject to distortion. Yet, if…

To blind or not researcher's identity has often been a topic of debate in the context of peer-review process for scientific publication and research grant application. This article reports on how knowing the name and experience of researchers/institutions influences the qualification of a proposal. We present our experience of managing the…