Please note that these guidance notes are for shortlisted applicants, but may be of interest for those currently preparing outline applications.
Ten common reasons why full applications to the NIHR HTA Commissioning committee can fail.
Thank you for applying to the HTA Programme, and well done for being shortlisted to submit a full application to the Commissioning committee. The committee wants to fund the commissioning brief because the question had been identified as an important one to the NHS. It is important to remember that funding is competitive, and that sometimes we have to turn down fundable proposals because other applications were better. The list below refers to a list of common problems which can result in applications failing at Stage 2, although most of the messages could apply to Stage 1 applications, too. They are intended to help you to get the best out of the HTA Programme.
Over-ambitious recruitment
Recruitment is often the most difficult area for determining the success of a clinical trial, and our committee will need to be convinced that you have thought through your recruitment issues well. Pre-application work identifying how many patients would be available in your collaborating centres, what proportion would be eligible and what proportion are likely to accept randomisation to the proposed interventions is always helpful. The committee likes to know how robust the assumptions used are, and for the applicants to identify areas of uncertainty. The HTA has a long track record of commissioning and monitoring trials, and it is aware of the probable conversion rate from invitations to participate to enrolment in the trial. On some occasions this type of information can be sourced from the literature, but it is usually prudent to recognise that these parameters are an estimate. Increasingly the committee is requesting feasibility studies or internal pilots to reduce uncertainty before applicants’ progress to the main phases of their trial. These studies can be integrated into an application for a full trial, but clear success criteria for progression should be identified. You may have been given some steer on this in the feedback from the committee.
Study just too small
We see a large variation in the sample sized posed by different applicants for the same brief. Some seem to have a zero missing from the sample size calculation. This is often due to assuming unrealistically high control event rates, looking for an implausibly large treatment effect, failing to take drop outs into account, failing to take clustering effects into account, etc. Remember that in most instances trials commissioned by HTA are pragmatic trials, testing how interventions function in everyday life – we expect interventions to be less effective than in efficacy trials. Unrealistically large effect sizes and event rates that are implausible should be avoided.
Lack for clear writing and inconsistencies
Clear, simple writing is appreciated by committee members and referees. Applications with overly complicated designs and descriptions can often obscure important aspects. Similarly, lack of care through sloppy mistakes such as typos and numbers inconsistently quoted in different parts can give the impression of a rush job which can detract from the stronger aspects of the application. Health economics and qualitative aspects are two aspects that are commonly underspecified.
Not having all the right people in the team
Conducting a successful pragmatic clinical trial, evidence synthesis, qualitative research, modelling or a diagnostic test study requires a combination of content expertise and methodological skills. Too often we see an excellent proposal with key people missing from the team. Conversely, a long list of applicants like a football team can be equally unconvincing if it is not clear what exactly all those mentioned are going to do. Typically a good team includes clinical content experts, methodological experts and those who are used to delivering the study type – all three components need to be present. Team members may be multidisciplinary or multi-sectoral. Methodologists could include statisticians, health economists or qualitative researchers as appropriate. Clinical Trial Units have been funded by the HTA Programme to support high quality applications, and it is important that your engagement with them is convincing rather than tokenistic in order to reassure the committee that appropriate design and data management skills are incorporated. If you do not have a CTU on board, then you will need to demonstrate in your application how and where those aspects typically delivered by a CTU will be dealt with. Whilst patient and public involvement (PPI) may not always be needed for all types of research, it is always relevant for HTA trials. Many PPI sections on the application are unconvincing to our consumer referees and committee members.
Drifting off the commissioning brief
You may have always wanted to do a particular study, but remember that the Commissioning committee undergoes a lengthy and open process to determine its NHS priorities and to then develop them into a commissioning brief. Sometimes, it is justifiable to explain why a different study design is more suited to the question than the one suggested in the brief, but it is almost always not a good idea to change the question in the brief.
Be open about problems and how you will address them
Our peer review process is very rigorous. If there are threats to the success of your study, try and identify them openly along with plans of how you will reduce such risk. Otherwise our referees might conclude that they have not been thought through properly. If there is something to hide, we will find it.
Lack of clinical equipoise
The committee will be unconvinced that your study team is best placed to answer the study question if all your background statements and investigator track record in a field are all very one-sided.
Unconvincing depth of understanding of the clinical problem
Applications that demonstrate a good depth of knowledge and discussion of the clinical dilemma posed tend to stand out. Some methodologically strong applications from experienced clinical trial or evidence synthesis units may sometimes fail because clinical input has been unconvincing.
Respond fully to committee feedback
Feedback from the committee offers you a good chance to address those things which were considered important at stage 1. Respond to all the points, and do not be afraid of saying “we agree” or “we disagree for the following reasons”, explaining where the proposal has changed. The time scale for responding to comments is quite tight, so please make sure that your team is able to respond to them rapidly when the time comes.
Application not good value
Value for money is defined by the quality of the methods proposed, the relevance of the question to NHS, the potential for health gain (or financial saving), and the risk (or potential) that the applicants will deliver a completed study, on time and within budget. You will be surprised by the large variation in costs included our applications for similar sized studies that cannot be easily explained. So please justify your costs carefully and make sure that your split between direct research and indirect research costs is apportioned correctly. Conversely, trials that are quite low in costs often raise suspicions of lack of trial experience or deliberate undercutting in the hope of winning a proposal only to ask for a funded extension subsequently.