Guidance for public reviewers of research funding applications
This guidance for is public reviewers of research funding applications to: NIHR Invention for Innovation; NIHR Programme Grants for Applied Research; NIHR Policy Research Programme; and NIHR Research for Patient Benefit.
Support for reviewers
Thank you for accepting our invitation to review a research funding application.
This guidance document aims to help you:
- understand the task of reviewing
- successfully complete a review.
Please visit our website to access some additional resources for patients and the public including:
- Our approach to reward and recognition for public contributors
- How to claim fees and expenses for involvement activities
- How to submit your review using the NIHR Central Commissioning Facility (CCF) online Research Management System (RMS)
- Top tips on what to consider when reviewing research proposals.
Please contact us if you would like to receive a paper copy of this guidance document or any of the other documents listed above.
Yvonne Anderson, Assistant Programme Manager
NIHR Central Commissioning Facility (CCF)
Telephone (direct line): 020 8843 8041
How the review process works
Reviews are an important part of how funding decisions are made.
They can both:
- inform the decision-making process of research programme panels
- provide funding applicants with feedback including an assessment of the strengths and weaknesses of their application.
Fee and expenses
We offer a fee for successfully completing a review and you will be sent a claim form after we receive your review. For more information on fees and expenses please see the CCF’s public payment guide.
Confidentiality and conflicts of interest
Please be aware that research applications are confidential. This is because research is a competitive area and researchers want to protect their original ideas. So, it’s very important that you don’t discuss an application with anyone else, unless you check with us first. If you receive or print a hard copy to work from, we ask you to either shred the documents when the task is completed or send them back to us and we will reimburse postal expense through the expense form.
Please check with us if you think you may have a conflict of interest. For example, do you know anyone on the research team personally? Have you been involved in developing this application? For more information read: Confidentiality and disclosure: A guide for applicants, reviewers and commissioning panels.
What we need from your review
Your review is written from your perspective as a member of the public, informed by your knowledge and experience as a patient and/or carer and as an actual or potential user of services. The aspects of an application that we particularly value your comments on are:
- Is the proposed research important to patients, service users or carers?
- Is the proposed research looking for outcomes that are important to patients, service users or carers?
- Are people likely to be willing to take part in the research?
- Is the plain English summary of the funding application easy to understand? Could it be improved?
- How appropriate are any plans for patient and public involvement (PPI) in the research application? What difference will it make?
- How could the researchers improve their plans for PPI in their research?
- How will the researchers make sure the findings of the research reach the general population?
For some examples of reviewer comments, please see section 8: What makes a good public review? Remember, we are getting a range of reviewers to comment on each application, including those with professional and scientific expertise.
A research funding application can be anything from under 50 to well over 100 pages in length and it often includes annexes, supporting documents, CVs, charts and letters of support.
Before you complete the online review form you will have to read and find your way around the application. How you approach this task is up to you. Here are a few tips from experienced public reviewers:
- If you start by reading the plain English summary it can give you a useful overview of the application and ideas about what to look out for elsewhere.
- Applications can seem dense and impenetrable. It can be helpful to break down the review task into smaller chunks over a number of days. This gives you thinking time too. You may need to read some or all of the application several times, before and whilst you are completing the review.
- It’s impossible to say how long it takes to complete a review. Like most things, the more reviews you do the easier and quicker it gets to navigate through the documents and the process. However, some applications will just be more complex than others.
- You could spend hours looking for information and exploring the things that an application makes you realise you don’t know. But don’t lose sight of the fact that we have asked you to do the review because of your personal knowledge and experience as a patient, service user or carer.
- It may help you to think of yourself as a ‘critical friend’. Someone who is encouraging and supportive and who comments honestly and constructively about weaknesses and problems, as well as strengths and successes. Style and tone are important. Significant comments can be missed or dismissed because of an apologetic or a belligerent sounding remark.
- Single word answers are not helpful. Panel members who will read and use your review don’t just want to know what you think. They also want to know the reason or reasons why you think it. For example, ‘On the basis of my experience as a carer of a person with motor neurone disease, I would suggest that ….’, or ‘In the absence of any budget for PPI in this application it is difficult to imagine how ….’.
Completing the review form
The online form is separated into sections, which correspond to the selection criteria used by the panel. Under each section there are a number of prompts to help you complete the form. They are designed to help you think about assessing the funding application from a patient and public perspective. A template of the public review form is available from our website.
To complete the form, you can type your comments directly into the online form, although some people prefer to work offline and then copy and paste into the form.
Once you have inserted your comments into each section, you’ll need to rate each section. Your rating (i.e. Excellent, Good, Fair, Poor, Don’t know) should reflect the comments you have made. You’ll then be asked to score the application overall in line with the scoring guide. When you have completed all the sections, you will be able to submit your review.
Please note that you do not have to complete the form in one session: you can save your review and return to it later. For step-by-step guidance to completing the online form, please refer to our RMS User Guidance.
If you encounter any difficulties, please contact us (see section 1: Support for reviewers) and we’ll do our best to help.
Every application is different, and therefore every review will be different. This section provides some examples of public reviewer comments from previous reviews and summarises what is particularly helpful or unhelpful about them. Please note that the examples do not cover all the different sections of the review form.
i) Comments about whether you think the research asks an important question – as well as why you think this
Example: “As a mother of children with asthma, I consider that the project reflects real issues and will test a very practical model. Attending 6 monthly reviews at a GP's can be inconvenient and time consuming. Frequently patients see different nurses or GP's - this model offers an opportunity to build up a relationship with an accessible health adviser.”
Why is this a good review?
- The comments are based on personal experience
- It highlights the importance of the issues
- It gives a practical view
Find out more in our guidance on how to analyse the research question in an NIHR funding application.
ii) Comments about whether the researchers are measuring the right outcomes
Sometimes researchers may choose to measure changes that aren’t so important to patients. For example, researchers may propose to measure the impact of a treatment on people’s physical health but not whether people also experience a better quality of life. It would be helpful if you can comment on what you think is important to measure.
Example: “The reason women (or any patients) seek medical help is that the quality of life is being reduced by the burden of disease, or they fear it will be. This study seems to be concerned about measuring the amount of blood loss rather than the impact heavy menstrual bleeding has on the lives of the women concerned. It would be improved vastly if they also added some impact measurements such as days off work, days/time feeling ill, impact on social and family life."
Why is this a good review?
- It describes what is important to measure from the perspective of people affected
- It clearly identifies a gap in the research application and suggests how the study could be improved
iii) Comments about whether you think the research would work in practice
It might help to think about whether you would agree to take part in the research and why you would say yes or no.
Example: “Interviews with bereaved carers could be an invaluable source to help understand the patient's experience, but a great deal of care will need to be taken over the timing of this discussion. If it takes place too soon after the death, the views of the carers may be significantly different to those obtained a few months later, leading to either a more positive or negative view of the patient’s care. As a carer who lived in a different area to my deceased mother, how would I have been traced to participate in the research?”
Why is this a good review?
- It clearly identifies the challenges involved in working with bereaved carers
- It questions the practicality of researchers getting access to required carers
- It is based on personal experience
iv) Comments about impact and dissemination
How do the researchers plan to publicise the findings of their work? For example, do the researchers plan to tell the people who have taken part in their study about the results? Will they be writing an article for a patient organisation newsletter or website, or are they only planning to publish an article in an academic journal?
Example: “The findings should be discussed with a group of patients to help develop recommendations for implementation (if successful) that reflect the patient experience and perspective. It might be possible to do this on an internet discussion forum on a website such as Asthma UK's.”
Why is this a good review?
- It provides positive, practical suggestions for publicising results to patients, carers and service users.
v) Comments about the patient and public involvement in the proposal
For example, have the researchers talked with any local patients’ or carers’ groups about their plans? If they are involving patients on steering groups, have they budgeted for their travel expenses and payment for their time?
Example 1: “The PPI is disappointing. Simply using groups to ‘trawl’ for information is not involvement. Prostate groups in the UK are some of the most advanced male cancer groups. The skills they have in all aspects of this study are not being utilised in the best way. Simply put, a bit of consultation and a fait accompli of the finished article is not involvement. Poorly thought out and sad to see…”
Why is this NOT a good review?
- While this comment highlights the lack of PPI, it does not provide suggestions on how the research team could improve the proposed PPI.
Example 2: “Having read the application it is clearly evident that the research team have fully engaged with patients and carers when developing this application. They have also entered into dialogue with patients and charities to identify important areas for discussion and utilised patients’ technical skills where appropriate. Please could the research team think about the following aspects:
- Is there a specific experienced member of the research team responsible for co-ordinating, supporting and delivering patient and public involvement activities?
- Is there an age limit for patients wishing to participate?
- Will patients have the option of being able to contact their specialist nurse by phone?
- If recruitment is not as anticipated and other recruitment centres need to be sought in the West Midlands or wider NIHR Cancer Research Network will it be necessary to seek further ethical approval and would this cause a significant time delay resulting in the trial having to be extended?
- Would the team consider asking patients if possible to keep a brief diary (which could form part of the patient information sheet and be attached to the back ), in order for them to be able to note any changes in their health or any queries that they might have while taking part...”
Why is this a good review?
- It is clear and detailed
- It highlights the strengths of PPI in the proposal
- It raises a number of questions for the research team and the panel to consider
What happens next
After you have submitted your review, you will receive an email from us acknowledging receipt of it and sending you a form to complete to claim a fee. You can access a copy of your review, saved as a PDF file, in the ‘Submitted Reviews’ section of ‘My Reviews’ in your Research Management System account.
We will read your review shortly after you have submitted it. We may provide feedback to you on the content of your review or contact you to discuss it. The most common reasons for this are that the review:
- has a lot of sections that are left blank or only has yes or no answers
- doesn’t provide an assessment of patient and public involvement
- includes information that makes the reviewer individually identifiable
- includes comments that are potentially offensive.
If you have any questions about your review after submitting it, please don’t hesitate to contact us (see section 1: Support for reviewers).
The review that you have provided, along with all other reviews, will be sent to the panel that then makes a funding recommendation about that application. We will contact you to let you know the outcome of that funding decision. But please be aware that for successful applications this could be some months after you have completed the review, as there are often many checks and conditions applied before an application becomes part of a binding legal contract.
THANK YOU for getting involved. We very much appreciate and value the time, skills and effort that members of the public contribute to our work. A big thank you is also due to the public contributors who helped us write the guidance.