Guidance for completing a review task (public contributors)
External peer reviewer assessment form guidance
This guidance is for public contributors who are completing a review of a research proposal for NIHR funding programmes (managed by the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC)). Review of research proposals requires the completion of an external peer reviewer assessment form and this guidance provides the questions and issues for you to consider so that you can complete this form.
Public contributors play a vital role in the process of funding health and care research. In completing this form you are fulfilling three functions. Firstly, you will be providing comment on a research proposal (application) that will inform the funding committee's decision-making process; secondly, you will provide guidance to the research teams about their proposed work; thirdly, you will be helping us to ensure that the proposed research is what is needed.
To support our decision-making we seek a variety of individuals to provide a range of opinions about the proposed research. The range includes a variety of professional backgrounds (including Clinicians, Methodologists and Academics, as well as NHS managers and others as appropriate) as well as a public perspective, which we have asked you to provide. Please answer only those questions which you feel you are able to answer based on your own personal understanding, experience or interest.
This guidance should be read alongside the reviewer assessment form as you complete it in our Management Information System (MIS). It explains what we would like you to focus on in the different sections of the form. If you require this assessment form in a word format or would like paper copies in a font size of your choice, please ask for this directly by contacting the person who invited you to complete the review. A useful glossary of public involvement and research terminology called Jargon Buster can be found on the INVOLVE website.
Once you accept to undertake a review, you will need to confirm that you will not disclose to any other person the fact that the applicant has applied for a research award, nor will you disclose the content of the application to any other person or use the information for any purpose other than providing a review of it. If you would like to discuss any aspect of your review with anyone else you must contact us first, to ensure that it will not present particular problems. For instance, sharing the task with organisations such as charities. When completing your review, you should not mention other applications where a decision on the funding of a study has not been made publicly available. If you have any direct comparisons to make between competing applications, please use the confidential comments box which will only be seen by the committee and not the applicants.
Your completed responses are considered confidential by the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC). Your anonymised responses to the questions, excluding the confidential section, will be passed to applicants to consider and respond to before the funding committee meeting. A copy of the anonymised reviewer comments will also be shared with other reviewers after the committee meeting has taken place. Any comments made in the confidential section of the form will only be seen by members of the funding committee but please be aware that these may be subject to requests for release under the Freedom of Information Act and Data Protection Act (2018). Please take account of the potential for requested release of confidential sections when providing your written comments.
Potential competing interests
Please check the names of the lead applicant and co-applicants before completing your review. If you are aware of any potential competing interests that you may have then please include them in this section. If you are at all unsure or have any queries regarding competing interests, then please contact us.
We believe that in order to make the best decisions, we should know about any competing interests that reviewers may have. We will not reject your opinion simply because you declare a competing interest, but we do need to know about it. An example of a competing interest could be that you know someone in the research team personally, or if you could gain financially if the research is funded.
Completing your review on the NETSCC MIS
NETSCC staff will register you on to the MIS and send you a password. To login to the system and complete your review, please go to the MIS and enter your login ID (your email address) and password and click “Log In”
If you haven’t used the MIS system before, please check your email inbox for an email from us containing your username and password. If you lose your password, please click on the ‘Forgot your password’ link and enter your email address as your Login ID. You will then be emailed a temporary password, which you can use to login and then change to something more memorable. If you are having problems accessing the system, please contact us.
- Once logged in, under ‘My Tasks’, click on the task ‘Submit External Review Comments’. Please do not click ‘View PDF’ as this is not the application, but will be the PDF your review will be pulled into.
- To view the application click ‘Application PDF’ under the Download Application section.
- On the left hand menu, click on ‘Confidentiality Agreement’
- Next, click ‘Competing Interests’ and enter your response on this page. Click ‘Save and Continue’ when you are finished.
- You will be taken to the ‘Review Questions’ page, you can also access this page from the menu on the left. Under the ‘Text’ section, click the ‘Add’ button next to each of the questions and write your comments (based on the guidance below), click ‘save and close’ and progress to the next question.
- Once you have answered all the questions according to the guidance below, click ‘Continue’ or click the ‘Score’ link on the left hand menu and enter your score and any confidential comments you have as per the guidance below. Click ‘Save and Continue’.
- On the ‘Review & Submit’ page, once you are happy you have completed the review, click on the ‘Submit’ button to finish the task.
Where to start?
The Summary (in plain English) is always a good place to start, you will find this under the ‘Case for Support’ section. This should give you a good background to help you understand the proposal.
There may also be additional documents included, such as study flow diagrams and reference material which may help explain the study.
Most proposals will be in two parts:
- The application form that contains the information you need about what the project is planning to do and why. There is also detailed information about the team and the finances as well as the methodology and study design.
- The detailed project description that follows a standard format and forms part of the proposal. You will also find a fuller account of the proposed project as well as more detail about the way the project will be done and its timelines.
Please be aware that you may need to read through the documents several times before completing the reviewer assessment form. You need only review the sections that you feel comfortable commenting on. You should look at both parts of the proposal in order to understand it properly.
If you need guidance or assistance, please contact the person who has approached you to undertake the review. Alternatively, you can contact a member of our Public and Patient Involvement (PPI) team: call 023 8059 9302 or email email@example.com
Completing the reviewer assessment form
The two main sections of the form are used in different ways:
Section 1: Review Questions
This will normally be shared anonymously with applicants prior to consideration by the funding committee, and applicants will be given the opportunity to respond to the issues raised.
Section 2: Score and Confidential Comments
This section remains confidential unless a Freedom of Information request is received.
Section 1: Review Questions
The following points are provided as a guide towards what you may wish to consider when answering the questions; however, we would be grateful for any feedback or other points that you would like to raise as a result of your personal experience.
Your responses to the questions in section 1 are normally shared anonymously with applicants prior to consideration by the funding committee. Applicants are then given the opportunity to respond to the issues raised by reviewers prior to a final committee decision.
If you have any questions or issues you think the applicants should respond to prior to consideration by the funding committee then please raise these in question 1.5. The text boxes will expand to accommodate your comments.
Issues for consideration by the committee (1.1)
When answering this question you may wish to consider the following points:
Importance, acceptability and potential impact of the proposed research
- Is there a clear reason for doing this research?
- Is the research looking at an issue that is important to patients and/or members of the public?
- Are the research findings likely to make a difference? E.g. to treatments or public services?
- Does this research overlap with other publicly available research that you are aware of?
Patient and Public Involvement (PPI)
- To what extent do you think that the relevant people or groups (i.e. patients, carers, or members of the public) have been appropriately involved in the development of the research proposal and in what capacity?
- If funded, to what extent do you think that the relevant people or groups will be involved in the conduct of the proposed research?
- Do the applicants appear to have considered the costs of ongoing public involvement in the study?
- Do you think that the circulation of the research findings to members of the public has been well planned? Is it appropriate?
- Please outline any opportunities, which you feel the team may have missed, to share what they have learned.
Key strengths (1.2)
- What are the key strengths of this proposal? Please give an overall assessment of the key strengths of the research proposal from your perspective.
Key weaknesses (1.3)
- What are the key weaknesses of this proposal? Please identify and summarise any aspects of the proposal that you perceive to be a weakness from your perspective.
You may wish to consider the following questions when preparing your response in sections 1.2 and 1.3:
Proposed research methods, recruitment and scientific quality
- Could the recruitment process work?
- Would you be happy to become a participant in the research? Would you be happy for a friend or family member to become a participant in the research?
- Are there any issues, which may be caused due to the setting or timing of any proposed intervention, and commitments, that are likely to impact on the daily lives of participants
- Do you think the research is likely to be doable? What are your reasons?
- Have the researchers chosen the right study group?
- Are there any changes that should be made to the research to make it better for patients taking part in it?
- If you feel that the study may not work or may have difficulties, because the researchers do not understand enough about the relevant patient or participant groups, it is very important that you highlight this in your response.
Research team, resources and research management
- Does the research team have an appropriate mix of skills (i.e. researchers from different disciplines, managers and professionals, patients and/or public members)?
- Are the resources requested appropriate for the proposed study?
- Does the proposal appear to be good value for money?
Does the plain English Summary give a clear explanation of the research? (1.4)
The plain English summary is intended for an interested audience, who are not necessarily specialists. The summary should be written at roughly the same level as an article in a newspaper. With this in mind, please comment on the following:
- Does the plain English summary give a clear explanation of the research?
- Does it help you carry out your review? If not, why not?
- Is the language used appropriate and clear? If not, where are the problems?
- Are scientific terms, abbreviations and jargon explained? If not, which terms need explanation?
- If this research is funded, the plain English summary will be published on a variety of websites without the rest of this application form. Could this plain English summary be used on its own to describe the proposed research? If not, what further information is needed?
Questions to applicants (1.5)
If you have any questions, thoughts or reflections that you would like to raise with the applicants prior to an assessment by the funding committee, please raise them in this section.
Applicants are given an opportunity to respond to your anonymised questions and these will then be provided to the funding committee.
You might want to consider the following:
- Is there anything that you would like the researchers to clarify?
- Is there anything that you think they should consider or reconsider?
- Is there anything that you think they should include or exclude?
Section 2: Score & Confidential Comments
If you wish to raise any issues in confidence then you can do so in section 2.1 ‘Comments field’. In section 2.2 we ask you to provide a summary score. If you wish to provide a justification for your score then please do so in section 2.1.
Comments field (2.1)
Please use this section to raise any other comments you would like to make in confidence from the applicants.
This section should be used if you are concerned that making a comment may identify you to the applicants. It would be most useful if the majority of your comments were made in prior sections as your feedback may help applicants to improve their application. If you are unsure about this or have any queries then please contact us.
Summary score (2.2)
Please provide a summary score that reflects your overall assessment of the proposed research
Scoring can often be challenging, particularly as you may be keen to see research into an important area go forward. It may be worth considering the following table when deciding on the score to give the proposal. Your score should reflect your overall assessment of the proposal. For more information about the funding process please see Appendix A.
|Score||Description of application||Suggested outcome||What this means from your perspective, including considerations around the delivery, PPI, conduct and dissemination of the proposed research.|
|6||Excellent||Proposed research can be funded as it stands||I have no concerns about this proposal.|
|5||Good||Proposed research can be funded with minor changes||The concerns that I have about this proposal could be easily corrected.|
|4||Good potential||There is much merit in this proposal, but it could be funded, perhaps after resubmission, with additional external support||The concerns that I have about this proposal could be corrected.|
|3||Some merits||There are significant weaknesses in this proposal, but these could in principle be addressed.||I have significant concerns but it may be possible for these to be addressed, although it would not be straightforward to do so.|
|2||Poor||Weak proposal||I have very strong concerns, which would be difficult to address, but the proposal does have some merit.|
|1||Extremely poor||Unsupportable proposal||I have very strong concerns that would be extremely difficult to address.|
Feedback from reviewers
We are always seeking to improve our processes, including the guidance and support we give to our reviewers. Please let us know if you have any comments or feedback about any aspect of this process.
Appendix A: The funding process
When considering stage 2 applications, NIHR committee members have the responsibility to take into account the views from all reviewers and other committee members to decide what research to recommend for funding.
Committee members are asked to score each application based on the view they have formed, and an average score is calculated for the proposal. The highest scoring applications will be recommended for funding. Those proposals that are deemed ‘important’ but have other issues, which may prevent them achieving a high score, can be designated as a ‘fund with change’. All proposals which are designated as a ‘fund with change’ are required to go through an iterative process to ensure that they have corrected all issues before they are taken further.
All funding committees include public members, and all stage 2 applications should also have an assessment by an appropriately experienced member of the public.
For information about individual NIHR funding programmes you can follow the links below:
- Efficacy and Mechanism Evaluation (EME) Programme
- Health Services and Delivery Research (HS&DR) Programme
- Health Technology Assessment (HTA) Programme
- Public Health Research (PHR) Programme
- Systematic Reviews (SR) Programme