Introduction
The following guidance is supplementary to the guidance provided in the completing the online application form section of the relevant programme guidance notes and should be read in conjunction with this guidance. The guidance below is aligned with the guidance provided to applicants for other NIHR programme funding and is intended to help applicants to NIHR personal awards think though the scope of any clinical trial, pilot study or feasibility study that is to be included within a personal award application. The guidance is particularly relevant to applicants thinking of including a feasibility study as part of a personal award application and is intended to make applicants think through the next stage of their research upon completion of the fellowship or other personal award.
Feasibility and pilot studies
The definitions of pilot and feasibility studies below have been agreed across NIHR research programmes and all NIHR research training schemes where project funding is included.
We expect that when pilot or feasibility studies are proposed by applicants as part of a research training award, the applicant is clear about the route the substantive study will take upon completion of the training award. This will include having an appreciation of which funding stream to apply to for the substantive study, and having a clear understanding as to the requirements of that funding stream in terms of what the preliminary study should have achieved.
Feasibility studies
Feasibility studies are pieces of research done before a main study in order to answer the question “Can this study be done?” They are used to estimate important parameters that are needed to design the main study. For instance:
- standard deviation of the outcome measure, which is needed in some cases to estimate sample size;
- willingness of participants to be randomised;
- willingness of clinicians to recruit participants;
- number of eligible patients; carers or other appropriate participants;
- characteristics of the proposed outcome measure and in some cases feasibility studies might involve designing a suitable outcome measure;
- follow-up rates, response rates to questionnaires, adherence/compliance rates, ICCs in cluster trials, etc.
- availability of data needed or the usefulness and limitations of a particular database; and
- time needed to collect and analyse data.
Feasibility studies for randomised controlled trials may not themselves be randomised. Crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study.
If a feasibility study is a small randomised controlled trial, it need not have a primary outcome and the usual sort of power calculation is not normally undertaken. Instead the sample size should be adequate to estimate the critical parameters (e.g. recruitment rate) to the necessary degree of precision. It should be noted that an underpowered ‘exploratory trial’ is not the same as a feasibility study and is unlikely to be funded as part of a research training award.
The very nature of feasibility studies means that they are relatively high risk. On the one hand the end result may be to confirm that a full trial is not feasible and on the other hand, even if shown to be feasible, another funder may not be interested in supporting a full trial (because perhaps either the clinical question is not sufficiently important or there are ongoing trials in the area already in the portfolio). Applicants need to consider the likelihood of other NIHR funding programmes or other funders being interested in supporting a full trial in the future when including a feasibility study in a research training award application. The feasibility study should also represent a high quality training vehicle for the applicant around clinical trials and research training more widely.
Pilot studies
Pilot studies are a version of the main study that is run in miniature to test whether the components of the main study can all work together. It is focused on the processes of the main study, for example to ensure recruitment, randomisation, treatment, and follow-up assessments all run smoothly. It will therefore resemble the main study in many respects, including an assessment of the primary outcome. In some cases this will be the first phase of the substantive study and data from the pilot phase may contribute to the final analysis; this can be referred to as an internal pilot. Or at the end of the pilot study the data may be analysed and set aside, a so-called external pilot.
Feasibility and pilot studies: which programme should I apply to?
Before deciding on which programme to apply you should first consider whether you want to apply for a personal research training award or funding to undertake a specific piece of research. NIHR personal research training awards e.g. Fellowships are not project or programme grants and any trial or feasibility study included in a training award must fit within the scope of what the award is designed to achieve (i.e. does it represent a suitable training vehicle for the trainee?) If you are looking for funding to undertake a particular piece of research then funding from one of NIHR’s research programmes may be a more appropriate route. Before submitting an application, applicants should consider the eligibility criteria for funding and which programme is most suitable.
Feasibility studies within a NIHR Research Training Award
Clinical trials are expensive and the chances of successful completion are improved if it can be shown beforehand that key elements (such as the ability to recruit patients) are feasible before the main study starts. NIHR will therefore fund such feasibility studies which are investigations carried out before a main study in order to answer the question “Can this study be done?” The research plan for a feasibility study should therefore contain a brief outline of the proposed main study and a list of the ‘uncertain’ important parameters that are needed to design the main study, as described below.
The Research Plan section of the application form should include:
A brief outline of the intended main trial.
Some of these details will of course depend on the results of the proposed feasibility research but a key part of evaluating the value of a feasibility study is whether or not a full trial is likely to get funded. You should therefore briefly describe as far as you can what the main trial would look like. This might include (if they are known), whether it’s an individual patient randomised or cluster trial, the number of arms, the inclusion criteria, the nature of the intervention and of the comparator in the control group, the primary endpoint, and the possible range of clinical sites from which patients would be recruited.
A list of the parameters which the feasibility study intends to clarify or estimate.
These may include:
- the number of eligible patients, carers or other appropriate participants;
- an exploration of different methods of identifying/recruiting patients;
- the willingness of clinicians to recruit and randomise participants;
- the willingness of participants to be randomised;
- the practicality of delivering the intervention(s) in the proposed setting(s);
- variation in use or delivery of the intervention in each setting;
- acceptability of the intervention to the users;
- standard deviation of the outcome measure, which is needed in some cases to estimate sample size;
- follow-up rates, response rates to questionnaires, adherence/compliance rates, ICCs in cluster trials, etc;
- availability of data needed or the usefulness and limitations of a particular database;
- the time needed to collect and analyse data;
- exploring the opportunities for PPI (patient and public involvement) in the research design and its subsequent conduct.
In effect the research plan should describe which parameters are to be estimated and how these will be investigated.
A feasibility study does not necessarily need to include the following:
- a randomised design: the design will be determined by how it is proposed to reduce the uncertainty in the parameters described above
- an evaluation of the outcome of interest: that is left to the main study
- a primary outcome: if a feasibility study involves carrying out a small randomised controlled trial it is for the purpose of evaluating/testing trial processes not the intervention
- the usual sort of power calculation: the sample size should be adequate to estimate the critical parameters (e.g. recruitment rate) to the necessary degree of precision.
Clinical trials training as part of an NIHR Research Training Award
Anyone proposing to include a clinical trial as part of a research training award needs to bear in mind the primary purpose of any award is to provide an excellent training experience for the trainee. This is achieved through conducting a relevant and high quality research project or programme complemented by formal training, placements, research visits, collaborations, supervision and/or mentorship. With this in mind it is important that applicants proposing to include a clinical trial think through the wider aspects of clinical trials training and the skills 4 and experiences they need. The award as a whole must represent a high quality training vehicle for someone who wants to make a step change in their trajectory towards becoming a future health research leader competent in the design and conduct of clinical trials.
NIHR does not dictate what the training within a fellowship should be or how it is delivered as it should be bespoke to an individual’s needs and requirements. However it is very important that the trainee experiences all elements of clinical trials from idea to dissemination. It is particularly important for example to ensure training in patient and public involvement and health economics (where needed) is included early in the lifecycle of a training award to ensure these elements can be utilised right from the inception of the study.
Below is a list of key skills that applicants should consider when putting together an application focussed around clinical trials. The list is intended to cover all aspects of clinical trials that future health research leaders competent in clinical trials should be knowledgeable in and it may not be necessary for a training award to encompass all these elements depending on the skills and experience the applicant already has.
- Evidence based medicine and critical appraisal of clinical trials
- Systematic reviews
- Basic statistics for clinical trials including power calculations
- Clinical trial design and protocol design
- Complex interventions and intervention development
- Randomisation
- Blinding
- Governance including; GCP, regulatory requirements, ethics.
- Recruitment
- Data collection, processing and management
- Data analysis
- Trial reporting, dissemination and impact
- Patient and public involvement
- Outcome measurement
- Health economics
- Priority setting and question development
- How to reduce bias and research wastage
- Funding for pilot and feasibility work
- Multidisciplinary working encompassing leadership, networking, and collaborating
- Ideas generation
In order to experience all the elements of a clinical trial as described above it may also be beneficial to gain exposure to the following as part of any research training award:
- Trial development groups
- Trial management groups
- Data monitoring and ethics committees
- Trial steering groups
- Dissemination meetings
Another important consideration when thinking through a research training application involving a clinical trial, particularly at PhD level, is whether the proposal will meet the requirements of a PhD to be ‘new and original research’ given that an RCT is highly multidisciplinary teamwork. A suitable project at PhD level for someone wanting to develop their career in clinical trials may not involve actually undertaking a clinical trial or feasibility study. It may be more appropriate and beneficial to consider projects based more around 5 the methodology of clinical trials, for example a PhD focused on a particular theme across several clinical trials such as recruitment, retention, qualitative studies in advance of a trial, obtaining consent, data analysis, or novel trial designs.
It is also important to link with and get input from other parts of NIHR when putting together an application focused on clinical trials. For example early discussions with the Clinical Research Network, Research Support Service and relevant Clinical Trials Unit (CTU) are strongly advised before starting an application. To order to help you identify a suitable CTU that is potentially willing to collaborate with you and support your training and development, the UKCRC Registered Clinical Trials Unit Network has a resource finder where you can search for a CTU based on various criteria. You are able to search for CTUs that are interested in supporting fellowships and other research training award applications and also search based on the disease area, study type and methodological expertise of the CTU.
Points to consider when evaluating established interventions
Many types of intervention such as cognitive behavioural therapy (CBT), exercise and outreach have been shown to be widely effective. Even so NIHR programmes, frequently receive applications for further evaluations of the effectiveness of such approaches or variants of them in different populations and for different indications. In judging such applications, three common questions arise:
Is a new trial in a different target population justified?
A common type of proposal is to evaluate (through a trial or trial feasibility study) the effectiveness of an intervention for a condition for which it has already been tested, but in a new population – for example, an exercise intervention in young people, old people or ethnic minorities. Another common application is for funding to evaluate the intervention in a familiar population that has a physical condition not previously included in previous research – prostate cancer, multiple sclerosis, frequent attendance in primary care and so on. These may be important research questions but a panel will reasonably ask if results suggesting effectiveness in previous research can be extrapolated to the new population or condition so that further research is not needed. After all, the popularity of some interventions such as CBT and exercise encouragement resides in being flexible therapies defined by some general principles, the detailed content of the intervention often being tailored to individual need during therapy.
When submitting applications for evaluating an established intervention in a new target population or condition it is therefore important to identify why and how the new target is different from others that have already been researched. An application justified simply by stating that the intervention has never been tested in the proposed target population is unlikely to be successful if that is the only rationale. A case needs to be made that the new target population or condition has important differences that make extrapolation from previous work inadvisable - for example that the new population has been shown to have a different response to other therapies in other studies, or the new physical condition poses challenges that have not been addressed in previous trials. In short, it is not the absence of evidence that best justifies new studies but the distinctiveness of the target population.
Is a trial of a new variant of an established intervention justified?
The second type of study that is frequently received by the programme is testing of another therapy based upon modifying the form or content of an established one. There are two issues for panels to consider here.
First, proposals may not change the content of therapy but propose different formats for delivery – for example using computerised CBT, smartphone apps or therapists from different disciplines to deliver the intervention. In this situation it is unlikely that the new variant would have considerably greater effectiveness than the conventional therapy and the rationale is usually that cost-effectiveness can be increased by the new format. In many instances further research may not be justified: it might seem a reasonable inference that if, say, chronic obstructive pulmonary disease (COPD) nurses can deliver CBT effectively then cancer nurses or health visitors can too. But if a trial is proposed, the required non-inferiority design will need a large and therefore expensive study and applicants need to bear in mind that the cost of any trial might be judged as outweighing the potential incremental benefit to be achieved. Alternatively, new formats need to be justified by evidence that they are likely to increase coverage or retention in therapy and will therefore be more effective at a population level.
Second, a new variant is sometimes proposed because it is argued that an existing generic intervention does not adequately treat the population or the condition under consideration. Examples might include modification to respond to specific symptoms not otherwise addressed or to specific features of the target population. Since, as noted above, many of these interventions are characterised by their flexibility, a panel will reasonably ask if the proposed variant is really new or simply codifies what a competent therapist would do anyway. Applications for variants of established interventions therefore need to make a strong case either that the new therapy is likely to be considerably more effective (or cheaper) than the existing one if the latter is delivered according to accepted standards.
What facet of the intervention is being evaluated?
The exact active ingredient in many interventions is not well understood. For example, there are two other components of the response to talking therapies from which the CBT effect needs to be differentiated. One is the non-specific effect of concerned attention, represented for example in the frequency and number of sessions. The other is generic therapeutic effects - the therapist’s skills and experience, the strength of the therapeutic alliance and so on. CBT is a limited and expensive resource in the NHS and applications will need to demonstrate that any effect demonstrated by the proposed intervention can reasonably be attributed to CBT and not to something that could be delivered more cheaply and just as effectively by other means.
Useful References and Resources
- Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract 2004, 10:307-12.
- Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology 2010, 10:67.
- Julious SA. Sample size of 12 per group rule of thumb for a pilot study. Pharmaceutical Statistics 2005, 4:287–291.
- The MRC provide a course entitled How to be a good Chief Investigator
- NIHR video The role of Clinical Trials Units in developing an NIHR funding application
- Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, Bond CM. PLoS ONE 2016, 11(3): e0150205.