ERIC Number: EJ1318661
Record Type: Journal
Publication Date: 2021-Jul
Pages: 12
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0958-2029
EISSN: N/A
Available Date: N/A
Does Reviewing Experience Reduce Disagreement in Proposals Evaluation? Insights from Marie Sklodowska-Curie and COST Actions
Seeber, Marco; Vlegels, Jef; Reimink, Elwin; Marusic, Ana; Pina, David G.
Research Evaluation, v30 n3 p349-360 Jul 2021
We have limited understanding of why reviewers tend to strongly disagree when scoring the same research proposal. Thus far, research that explored disagreement has focused on the characteristics of the proposal or the applicants, while ignoring the characteristics of the reviewers themselves. This article aims to address this gap by exploring which reviewer characteristics most affect disagreement among reviewers. We present hypotheses regarding the effect of a reviewer's level of experience in evaluating research proposals for a specific granting scheme, that is, scheme reviewing experience. We test our hypotheses by studying two of the most important research funding programmes in the European Union from 2014 to 2018, namely, 52,488 proposals evaluated under three funding schemes of the Horizon 2020 Marie Sklodowska-Curie Actions (MSCA), and 1,939 proposals evaluated under the European Cooperation in Science and Technology Actions. We find that reviewing experience on previous calls of a specific scheme significantly reduces disagreement, while experience of evaluating proposals in other schemes--namely, general reviewing experience, does not have any effect. Moreover, in MSCA-Individual Fellowships, we observe an inverted U relationship between the number of proposals a reviewer evaluates in a given call and disagreement, with a remarkable decrease in disagreement above 13 evaluated proposals. Our results indicate that reviewing experience in a specific scheme improves reliability, curbing unwarranted disagreement by fine-tuning reviewers' evaluation.
Descriptors: Foreign Countries, Evaluators, Interrater Reliability, Research Proposals, Individual Characteristics, Experience
Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: European Union
Grant or Contract Numbers: N/A
Author Affiliations: N/A