Peer review is integral to the evaluation of grant proposals. Reviewer perceptions and characteristics have received limited study, especially their associations with reviewers' evaluations. This mixed methods study analyzed the unstructured comments of 270 experienced peer reviewers after they scored proposals based on mock overall evaluations…

In the early months of the COVID-19 pandemic, many research funding organizations were faced with the choice of suspending their peer review panels, or else continuing their decision-making processes virtually. Although seen part of a longer drive to make peer review more cost and time efficient as well as to combat climate and sustainability…

Diverse efforts are underway to reform the journal peer review system. Combined with growing interest in Open Science practices, Open Peer Review (OPR) has become of central concern to the scholarly community. However, what OPR is understood to encompass and how effective some of its elements are in meeting the expectations of diverse communities,…

Peer review has developed over time to become the established procedure for assessing and assuring the scientific quality of research. Nevertheless, the procedure has also been variously criticized as conservative, biased, and unfair, among other things. Do scientists regard all these flaws as equally problematic? Do they have the same opinions on…

As part of the shift towards a more equitable research culture, funders are reconsidering traditional approaches to peer review. In doing so, they seek to minimize bias towards certain research ideas and researcher profiles, to ensure greater inclusion of disadvantaged groups, to improve review quality, to reduce burden, and to enable more…

Previous studies of the use of peer review for the allocation of competitive funding agencies have concentrated on questions of efficiency and how to make the 'best' decision, by ensuring that successful applicants are also the more productive or visible in the long term. This paper examines the components of feedback received from an unsuccessful…

There has been a recent increase in attention toward the proper targets of evaluation in science funding policy. Specifically, some claim that we should 'fund people, not projects' to allow for increased autonomy for researchers. Critics argue that this movement unduly opens room for biases against several marginalized groups of scientists. In…

Despite the surging interest in introducing lottery mechanisms into decision-making procedures for science funding bodies, the discourse on funding-by-lottery remains underdeveloped and, at times, misleading. Funding-by-lottery is sometimes presented as if it were a single mechanism when, in reality, there are many funding-by-lottery mechanisms…

The way research is, and should be, funded by the public sphere is the subject of renewed interest for sociology, economics, management sciences, and more recently, for the philosophy of science. In this contribution, I propose a qualitative, epistemological criticism of the funding by lottery model, which is advocated by a growing number of…

In the current higher education landscape, universities are facing expanding requirements beyond teaching and research. Evaluation methods must evolve accordingly to prevent universities from facing development dilemmas. Current mainstream evaluation methods primarily emphasize the research domain, often failing to holistically capture a…

This article assesses the risk of two COVID-19-related changes necessary for the expert review of the REF2021's Impact criterion: the move from face to face (F2F) to virtual deliberation; and the changing research landscape caused by the COVID-19 crisis requiring an extension of deadlines, and accommodation of COVID-19-related mitigation. Peer…

Reviewers are humans and might be affected by cognitive biases when information overload comes into play. In fact, no amount of scientific training will completely mask the human impulses to partisanship. And the consequence is that authors may receive incorrect editorial decisions in their submissions to peer-reviewed journals. For instance, the…

This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements. Empirical evidence indicating that there is no consensus or more than one consensus would offer an explanation for the "disagreement…

The European Research Council (ERC) receives many high-quality applications, but funds only a few. We analyze how members of ERC review panels assess applications in the first, highly competitive step of evaluations for ERC Starting and Consolidator Grants. Drawing on interviews with ERC panel members in different fields, we show that they adopt a…

Securing research funding is essential for all researchers. The standard evaluation method for competitive grants is through evaluation by a panel of experts. However, the literature notes that peer review has inherent flaws and is subject to biases, which can arise from differing interpretations of the criteria, the impossibility for a group of…