Guidance for completing a review task (professional contributors)
External peer reviewer assessment form guidance
This guidance is for external peer reviewers who are completing a review of a research proposal from a professional perspective for NIHR funding programmes (managed by the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC)). Review of research proposals requires the completion of an external peer reviewer assessment form.
In completing this form you are fulfilling three functions. Firstly, you will be providing an appraisal of the research proposal (application) that will inform the funding committee's decision-making process; secondly, you will provide guidance to the research teams about their proposed work; thirdly, you will be participating in our quality assurance process to ensure that the proposed research is important and timely, scientifically robust and offers good value for money.
We will be seeking comments from a range of individuals, both from professional backgrounds and members of the public, to gain a range of opinions as to the merit of the proposed research. Please answer only those questions which you feel able to, given your own areas of knowledge and expertise.
Once a funding decision has been made and communicated to applicants, the decision along with the open and anonymised comments from all the reviewers of the proposal will be shared with you.
Please feel free to use as much or as little space as you wish as the boxes provided for your response will expand to accommodate your text.
Once you accept to undertake a review, the applications are shared with you in confidence to assess the quality and relevance of the research proposal. You will need to confirm that you will not disclose to any other person the fact that the applicant has applied for a research award, nor will you disclose the content of the application to any other person (including work colleagues) or use the information for any purpose other than providing a review of it to the NIHR. If you wish to seek input from a colleague then you need to obtain permission from NIHR funding programme staff before sharing any details of the application or applicants. When completing your review, you should not reference other applications, or disclose any contents contained therein where a decision on the funding of a study has not been made publicly available. If you have any direct comparisons to make between competing applications, please use the confidential comments box, which will only be seen by the committee and not the applicants.
Your completed responses are considered confidential by the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC). Your anonymised responses to the questions, excluding the confidential section, will be passed to applicants to consider and respond to prior to the funding committee meeting. A copy of the anonymised reviewer comments will also be shared with other reviewers after the committee meeting has taken place. Any comments made in the confidential section of the form will only be seen by members of the funding committee, but please be aware that these may be subject to requests for release under the Freedom of Information Act and Data Protection Act (2018). Please take account of the potential for requested release of confidential sections when providing your written comments.
Potential competing interests
To make the best decisions on proposals for research, we should know about any competing interests that reviewers may have. We have already conducted our own checks to rule out any institutional conflicts (i.e. the applicant or co-applicants and the reviewer based at the same organisation). But it is difficult for us to know about complex professional and interpersonal relationships between individuals working in a similar field of interest. The BMJ Editors define a competing interest as existing when a professional judgement concerning a primary interest (such as patients’ welfare or the validity of research) may be influenced by a secondary interest (such as financial gain or personal rivalry).
Please do check the list of ALL applicants and co-applicants in ALL of the applications you are being asked to review before starting the review, and if you have a concern about a potential conflict of interest then please do contact us for further advice prior to completing the review.
If you have a conflict of interest which we do not deem to be significant you may be asked to complete the review, if so, please provide details in the box provided in section 2.3. We will not reject your opinion simply because you declare a competing interest, but we would like to know about it.
Completing your review on the NETSCC MIS
NETSCC staff will register you on to the MIS and send you a password. To login to the system and complete your review, please go to the MIS and, if you have a password, enter your login ID (your email address) and password and click “Log In”
If you haven’t used the MIS system before and do not have a password, you will need to request a password. To do this, click on the ‘Forgot your password’ link and enter your email address as your Login ID. You will then be emailed a temporary password, which you can use to login and then change to something more memorable. If you are having problems accessing the system, please contact us.
- Once logged in, under ‘My Tasks’, click on the task ‘Submit External Review Comments’. Please do not click ‘View PDF’ as this is not the application, but will be the PDF your review will be pulled into.
- To view the application click ‘Application PDF’ under the Download Application section.
- On the left hand menu click on ‘Competing Interests’
- Enter your response on this page, and click ‘Save and Continue’.
- You will be taken to the ‘Review Questions’ page, you can also access this page from the menu on the left. Under the ‘Text’ section, click the ‘Add’ button next to each of the questions and write your comments (based on the guidance below), click ‘save and close’ and progress to the next question.
- Once you have answered all the questions according to the guidance below, click ‘Continue’ or click the ‘Score’ link on the left hand menu and enter your score and any confidential comments you have as per the guidance below. Click ‘Save and Continue’.
- On the ‘Review & Submit’ page, once you are happy you have completed the review, click on the ‘Submit’ button to finish the task.
For information about individual NIHR funding programmes you can follow the links below:
- Efficacy and Mechanism Evaluation (EME) Programme
- Health Services and Delivery Research (HS&DR) Programme
- Health Technology Assessment (HTA) Programme
- Public Health Research (PHR) Programme
- Systematic Reviews (SR) Programme
The review task
Section 1: Review Questions
The following points are provided as a guide towards what you may wish to consider when answering the questions; however, we would be grateful for any feedback or other points that you would like to raise as a result of your specialised knowledge, skills and experience.
Your responses to the questions in section 1 are normally shared anonymously with applicants prior to consideration by the funding committee. Applicants are then given the opportunity to respond to the issues raised by reviewers prior to a final committee decision.
If you have any questions or issues you think the applicants should respond to prior to consideration by the funding committee then please raise these in question 1.5. The text boxes will expand to accommodate your comments.
Issues for consideration by the Committee (1.1)
This section can be presented as ‘major’ (those issues which could prevent the application form being funded) or ‘minor’ (issues which could be resolved).
You may wish to consider the questions below when preparing your response (where applicable).
Importance and potential impact of the proposed research
- Does the research meet an identified need (e.g. a gap in the evidence base)? Does it meet the requirements as specified in the commissioning brief/specification document?
- Is the research question posed the most important question? If not why not? How could the question be improved?
- What impact is this research likely to have on clinical practice, improving health services and/or public health?
- What impact is this research likely to have on scientific knowledge?
- Does this research ‘overlap’ with other similar research that you are aware of?
- Does the proposal make a clear case for the research proposed with reference to the current evidence base?
Patient and public involvement
- Have patients, carers or members of the public been involved in the development of the research proposal? In what capacity?
- In what capacity will patients, carers or members of the public be involved in the implementation of the proposed research? Is this appropriate?
- Have appropriate resources been allocated to ensure effective patient and public involvement?
- Is the proposed research based on Intellectual Property (IP) developed by others? Does the applicant have access to use the existing IP? Who will own any future IP?
- Have the team considered Intellectual Property (IP) being generated by this study? (e.g. new or improved checklist; patient data; software tool; tool kit for patient care; database design; medical device, pharmaceutical drug, or application; equipment design; new data of corporate significance)
- Has the project team identified any potential to exploit developments and with whom?
- Do you believe there to be any additional opportunities to exploit or further develop the outputs from the proposed research?
- Is a commercial partner involved in the study? Is there a likely need for collaboration agreements or sub-contracts?
- Are there any changes that should be made to the research to make it better for patients taking part in it?
Key Strengths (1.2)
- What are the key strengths of this proposal? Please give an overall assessment of the key strengths of the research proposal.
Key Weaknesses (1.3)
- What are the key weaknesses of this proposal? Please identify and summarise any aspects of the proposal that you perceive to be a weakness.
You may wish to consider the following questions when preparing your response in sections 1.2 and 1.3:
Consideration of methods and scientific quality
- Is the proposed research feasible?
- Are the research methods robust?
- Will the research methods suggested provide answers to the research question(s)?
- Are the plans for data collection and analysis sufficiently detailed?
- Are any assumptions made? Are they reasonable?
- Is the study design appropriate?
- Are the research subjects and the methods of identifying them relevant and sufficient?
- Are the interventions adequately described and appropriate?
- Are the effects, measures or information sought appropriate and relevant to the study’s purpose?
- Are the outcomes and duration of follow up appropriate?
- Do the planned analyses appear reasonable?
- Where appropriate, is there a clear explanation and justification for the sample size and/or estimated recruitment rates?
- Is the sample size based on a believable effect size?
- Are there any changes that should be made to the research to make it better for patients taking part in it?
Research team, resources and research management
- Does the research team have an appropriate mix of skills (i.e. researchers from different disciplines, managers and professionals, patients and/or public members)?
- Is the role of each collaborator/co-applicant clear?
- Have the research team sought representation from relevant professional groups or practitioners required to inform the study design and delivery?
- Do the applicants demonstrate an adequate track record appropriate to their proposed roles?
- If the study involves more than one organisation, are the relationships between the various organisations clearly defined?
- Are the resources requested appropriate for the scale and methods of the proposed study?
- Where appropriate a) is there a realistic chance of recruiting the required number of participants within the time stated and b) if you had the relevant condition, would you be willing to participate as a patient in the study?
- Does the proposal represent good value for money? Does the total funding required (research grant and NHS costs) represent good value for money?
- Are the applicant’s estimates of the resource implications of the patient care costs (NHS Support and Treatment costs, including Excess Treatment costs) substantially over-estimated, substantially under-estimated or about right?
Ethics and governance
- Is the proposed research ethically sound?
- Are the study governance arrangements appropriate?
- Have the necessary approvals been obtained (or measures included to address this) from the relevant ethical and regulatory bodies?
- Have the applicants indicated that they are aware of the appropriate trial monitoring structures that need to be in place, and how they intend to to implement them? Eg Process for informed consent, a Trial Steering Committee (TSC), a Data monitoring and Ethics Committee (DMEC).
Plain English summary (1.4)
The plain English summary is intended for an interested audience, who are not necessarily specialists. The summary should be written at roughly the same level as an article in a newspaper. With this in mind, please comment on the following:
- Does the plain English summary give a clear explanation of the research?
- Does it help you carry out your review? If not, why not?
- Is the language used appropriate and clear? If not, where are there problems?
- Are scientific terms, abbreviations and jargon explained? If not, which terms need explanation?
- If this research is funded, the plain English summary will be published on a variety of websites without the rest of this application form. Could this plain English summary be used on its own to describe the proposed research? If not, what further information is needed?
Further information for researchers on how to write a plain English summary and what to include in a summary is available online at NIHR Make it clear.
Questions for applicants (1.5)
If you have any questions you would like to raise with the applicants prior to an assessment by the funding committee, please raise them in this section. These questions are sent anonymously to the applicant who is then given an opportunity to respond. The funding committee will see the applicant’s response to these questions.
Section 2: Score and confidential comments
If you wish to raise any issues in confidence then you can do so in section 2.1 ‘Comments field’. In section 2.2 we ask you to provide a summary score. If you wish to provide a justification for your score then please do so in section 2.1.
Comments field (2.1)
Please use this section to raise any other comments you would like to make in confidence from the applicants.
This section should be used if you are concerned that making a comment may identify you to the applicants. It would be most useful if the majority of your comments were made in prior sections as your feedback may help applicants to improve their application. If you are unsure about this or have any queries then please contact us.
Summary score (2.2)
Please provide a summary score that reflects your overall assessment of the proposed research
Scoring can often be challenging, particularly as you may be keen to see research into an important area go forward. It may be worth considering the following table when deciding on the score to give the proposal. Your score should reflect your overall assessment of the proposal. For more information about the funding process please see Appendix A.
|Score||Description of application||Suggested outcome||What this means from your perspective, including considerations around the delivery, PPI, conduct and dissemination of the proposed research.|
|6||Excellent||Proposed research can be funded as it stands||I have no concerns about this proposal.|
|5||Good||Proposed research can be funded with minor changes||The concerns that I have about this proposal could be easily corrected.|
|4||Good potential||There is much merit in this proposal, but it could be funded, perhaps after resubmission, with additional external support||The concerns that I have about this proposal could be corrected.|
|3||Some merits||There are significant weaknesses in this proposal, but these could in principle be addressed.||I have significant concerns but it may be possible for these to be addressed, although it would not be straightforward to do so.|
|2||Poor||Weak proposal||I have very strong concerns, which would be difficult to address, but the proposal does have some merit.|
|1||Extremely poor||Unsupportable proposal||I have very strong concerns that would be extremely difficult to address.|
Appendix A: The funding process
When considering stage 2 applications, NIHR committee members have the responsibility to take into account the views from all reviewers and other committee members to decide what research to recommend for funding.
Committee members are asked to score each application based on the view they have formed, and an average score is calculated for the proposal. The highest scoring applications will be recommended for funding. Those proposals that are deemed ‘important’ but have other issues, which may prevent them achieving a high score, can be designated as a ‘fund with change’. All proposals which are designated as a ‘fund with change’ are required to go through an iterative process to ensure that they have corrected all issues before they are taken further.
All funding committees include public members, and all stage 2 applications should also have an assessment by an appropriately experienced member of the public.