Published: 18 September 2019
Celebrating five successful years, Peer Review Week runs from 16 to 20 September 2019 with ‘Quality in Peer Review’ as its theme. This is timely for the NIHR Research on Research (RoR) team as we are currently exploring ways to enhance peer review in research funding. What better way to contribute to this celebration than by sharing some of the insight gained from our on-going research and more importantly acknowledging the incredible contributions from peer reviewers themselves? Not only are they spending valuable time reviewing for the NIHR, they are also contributing and participating in our on-going research.
Before we go straight into detail, let’s consider for a moment what we mean by peer review, knowing that although the concept is familiar to many of us, there is great variation in how it can be interpreted.
The role of peer review in research funding allocation
In a nutshell, peer review serves as a tool and is used in a range of ways to determine and evaluate the value, quality and merit of the research and its intended outputs, outcomes and impact. Behind the scenes of the decision to fund research, there is a substantial process of internal and external peer review. Peer review is used in this context to assess the quality of a research proposal based on the relevance of the research question, methodological design, make-up of research teams, and value for money.
Peer review is regarded as gold standard (some may argue it is simply the de facto system); it is the undisputed champion of decision-making. Organisations that fund research use peer review daily to support the decisions made by funding committees who have the responsibility of allocating scarce available funds to advance science. Peer review is complex and has a highly valuable contribution to make, yet a large body of evidence suggests that there are challenges associated with peer review.
What does this mean for NIHR?
The RoR team aims to provide broad relevance across the NIHR by conducting research on its own practice, helping to sustain the NIHR’s reputation for being innovative and evidence-based in our management processes of research.
From the ongoing work conducted by the RoR team and through the evidence from the realist synthesis, suggests the current peer review processes have high administrative costs, there is low reliability between reviewers, bias, and burden for all stakeholders involved. The realist synthesis also suggests that there is little evidence to guide funders like NIHR to suggest what we might do better to optimise the decision-making process. The evidence we generate will help to shape our commitment to continuous improvement relating to the role peer review takes in our decision-making processes.
So, what can we do?
Everybody wants an efficient and stable system that supports researchers and funders, but we need these solutions grounded in evidence. This is where RoR’s work adds value. Understanding current practices and alternatives to peer review, as well as identifying contexts in which alternative approaches are effective, will inform subsequent research into how funders like the NIHR can enhance their own processes.
Brief snapshot of RoR team’s work
We are conducting a realist synthesis of decision-making approaches to research fund allocation, contributing to the little empirical evidence on the effects of funding allocation peer review. Preliminary analysis of over 600 publications show only a handful of interventions measured the efficiency of peer review, or that have implemented slight or radical alternatives to the traditional peer review process and measured the effect of the changes (in terms of time or cost efficiencies or improved transparency).
However, these interventions seem to ebb and flow, therefore, conversations and discussion taking place during Peer Review Week are vital. Understanding quality and value enhances our understanding around the peer review processes and what alternatives have worked. We hope that this will encourage longer-term implementation of successful mechanisms to the decision-making process of research funding.
Our other research activities include:
- A survey with international funders to understand current funding practices
- Qualitative analysis of interview data to understand stakeholders’ expectations
- A review of the feedback given to applicants to understand what makes a good application
- A survey with applicants to understand the value of the feedback received
- An observational study of funding committees.
All the evidence has a clear and intended outcome: generate evidence to corroborate or dispute the current de facto stance of peer review. As Isaac Newton once said:
“I seem to have been only like a boy playing on the seashore, and diverting myself in now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me”
We would like to personally thank all those who give up their time to peer review for the NIHR - those who complete peer reviewer forms, members of the public, those who sit on prioritisation and funding committees and to those within NIHR who review on a daily basis. It’s not an easy task so from everyone in the RoR team, we want you to know we truly value your contributions and effort.
- Amanda Blatch-Jones, Senior Research Fellow, NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC), NIHR RoR team
(Contributors to this blog include RoR team members Alex Recio Saucedo and Katie Meadmore)
*RoR are a team of researchers based at the University of Southampton, working out of NIHR Evaluation, Trials and Studies Coordinating Centre. RoR@nihr.soton.ac.uk
The RoR team will be at the International Clinical Trials Methodology Conference on Sunday to Wednesday, 6 to 9 October 2019. If you would like to find out more about RoR come and visit them at their poster displays, or visit their website.
Other blogs from the ROR team include:
More information for external peer reviewers and public reviewers is available on the NIHR website.
During Peer Review Week, NIHR reviewers and committee members have been telling us about the topic; ‘Quality in Peer Review’ and what it means to them:
Judith Dawson, NIHR External Peer Reviewer and GP who has a background in commissioning and works as an adviser to a social care charity:
“It’s very important to me to have rapid access to high quality research findings in order to advise my patients on the best options for them, as well as understanding new directions in health research and provision. This is why I value the chance to work in peer review for NIHR. It means I can contribute to the review process and enable the perspective of a community facing clinician to be incorporated into health research proposals.
“Choosing wisely at a time of distress and illness, against a background of economic restraint is difficult. Without the perspective of peer reviewed, high quality research to inform that choice, it would be impossible.”
Louise van Wingerden, NIHR Public Reviewer:
“Being involved in reviewing research proposals allows me to use my personal experience of the UK health system to help researchers consider how their work affects the public. On several occasions I have been able to flag up concerns, both minor and major, within the proposals. This means that any insight I have gained from my own experiences, has, I hope, proved useful to the wider public. I enjoy reading the research proposals, which are often couched in academic language, and thinking about how I can help to improve them. This sometimes seems a little presumptuous – but very often there turn out to be areas within the proposals which benefit from a layman’s vision.”
Debbie Keatley, HTA Funding CET Public Committee Member:
“Done well, the views of a peer reviewer can shift the thinking on a proposal and turn a lukewarm response into a more positive one (and vice versa). Some reviewers have a skill in seeking out and exposing key points that may be deep within an application, like a metal detector.
“I think the best reviews are succinct without being curt, easy to follow and don't beat about the bush.
“Approaching a long application is a daunting feat; a well-written lay summary and peer reviews that are written with clarity and precision make my job so much easier. It helps me find my way into a proposal - it gives me an overview, as well as insights into any strong points or potential problems. A great review will not only identify these but may have suggestions on how proposals could be strengthened.
“I have read reviews that are bland or one sentence long; in my opinion these offer little or no support for decision-making.”
Gail Thornton, HTA Funding CET Public Committee Member:
“A good quality review highlights the positive aspects of an application and provides an intelligible and cogent critique of weaker elements, within the field of expertise of the reviewer. It will ask relevant questions of the applicants, which we might have been unaware of ourselves, to help explain, expand on and clarify areas of uncertainty, ambiguity or omission in the proposal and challenge assumptions and unsupported assertions. It should focus attention on areas which need response from applicants, in order to help us assess the quality and validity of the proposal.
“A high quality review will be written to fulfil two overarching aims: to guide and challenge the applicants to improve their proposal further, and to inform and facilitate the decision-making of committee members.
“An unhelpful review does not contain enough information, or the rationale for a reviewer’s opinion.”