Published: 03 November 2020
Ben Morgan is Assistant Director for the NIHR Research for Patient Benefit (RfPB) programme and NIHR Research Schools Research Programmes. He writes about assessing health research grant applications and whether the processes being used currently, lead to funding the best research.
I have always found it slightly odd that while research funders spend a huge amount of money, time and effort to ensure they fund the best research with a view to produce the best evidence, funders rarely spend much time to determine if the processes and mechanisms they use are well evidenced and optimal in supporting their aims of funding the best research. This landscape is changing and indeed the NIHR has a ‘Research on Research’ team which looks to improve how NIHR funds research.
Understanding the commissioning process
Within the Research for Patient Benefit (RfPB) programme we had a chance to review some large scale changes we made to our commissioning processes back in 2015, essentially adopting a two-stage commissioning process after operating a single stage process for nearly 10 years. Many research programmes use a two-stage process but given RfPB’s very high volume of, relatively smaller (up to £350k), applications it was believed when setting up the programme a single stage process would be more appropriate when trying to balance the fundamentals of speed of assessment and rigour of assessment.
Increasing the quality of stage 2 applications
We decided to change to a two-stage process to make the effort for the researchers applying and for the peer reviewers more commensurate with the size and scale of awards on offer. We also thought a chance to respond to feedback at stage 1 would increase the quality of the stage 2 applications. However, at the same time we did not want to extend the time from application to outcome because not only do funders (and researchers) want to fund the best research they tend to want to start it and realise the benefits as soon as possible.
Therefore we wondered whether any potential savings in effort for research teams and reviewers would be worth a longer wait for researchers to be informed if they were funded or not.
Cost-savings vs time-efficiency
We published a review of our findings on Plos One which showed that the two-stage process led to an increase in total applications submitted (22 per round more on average) and that there were considerable time and effort savings for peer reviewers with the reviewer burden falling by half. There were also considerable cost savings of about 30% associated with the two-stage process due to how we were able to operationalise the new process. However, the assessments and ultimately the outcomes to research teams did take longer – 74 days on average – under the two-stage process but only for applicants who made it to stage 2. Applicants who were declined at stage 1 discovered their outcome on average 79 days earlier compared to the single stage process. Therefore, the answer was the two-stage process is both faster and slower than the single stage process. We took the view that researchers would prefer this approach; if they were declined at stage 1 they found out earlier but if they were invited to stage 2, which had a 50% success rate, the additional wait for outcome would be worth it.
Application success rates
We were happy that process-wise, the two-stage process was more efficient but what about the quality of the applications? We were pleased to see that external peer review scores under the two-stage process were higher (5.6% on average) so it appeared that the feedback given at stage 1 did improve the quality of the stage 2 application. Interestingly, public review scores did not change between the two assessment processes – perhaps the feedback at stage 1 is more focused on areas such as methods, instead of patient and public involvement (PPI)?
Ultimately, we were pleased (and relieved) that the new process was more efficient, improved the quality of applications and seemed more proportionate for all involved. Had it not been we would have probably had to revert back to the single stage process quite quickly! But at least we know and are able to quantify the changes we made and make an informed decision on whether the changes were worth it or not.
The Research for Patient Benefit programme funds health, public health and social care research covering a wide range of health service challenges.