This document outlines answers to questions asked by applicants to the NIHR i4i & OLS RWE Programme. More information is available in our guidance for applicants and in our Supporting information for applicants.
Please be aware the OLS RWE call closed on the 27/03/2024. Please contact the OLS RWE team i4i-rwe@nihr.ac.uk if you have any further questions.
Eligibility
Could you clarify if EVA is part of this call or is it a pre-requisite to compete on this call?
A recommendation for early use in the NHS through NICE EVA is a pre-requisite for technologies submitting to this Call.
Where can I read more information about each technology that has been through EVA which is eligible?
See the full list of technologies which have been recommended for early use in the NHS through NICE EVA. To be eligible for this Call, the guidance must be published by Call close.
Does the Evidence Generation Plan (EGP) also have to be published (not just the EVA recommendation)?
Yes, to be eligible for this Call, the technology must be recommended for early use in the NHS through NICE EVA and the guidance must be published by Call close.
Will there be a requirement for products to have relevant market authoriation (MHRA etc) and be DTAC assessed (if digital) - and how will this be assured?
Yes, digital health technologies must be DTAC compliant by the time of use in the NHS as part of the project. Products must also have a CE mark or equivalent, if and as appropriate, with justifications provided in the application, which will be assessed by the committee.
I have a technology that has had health tech evaluation but needs real world evidence. Can they apply?
Only technology that has been recommended for early use in the NHS through NICE EVA may be included in this Call. See the full list of technologies which have been recommended for early use in the NHS through NICE EVA.
Can the analytical partner and technology partner be the same entity?
The aim for this Call is to form 3-way consortia with an independent analytical partner with experience in real world evidence study design, conduct and analysis leading the application. As such, the analytical partner and technology partners should be different entities.
Can an analytical partner be the same as the adopting site looking to lead, coordinate and host testing of the product?
The aim for this Call is to form 3-way consortia with an independent analytical partner with experience in real world evidence study design, conduct and analysis leading the application. As such, the analytical partner and adopting site should be different entities.
Can the analytical partner be a commercial entity? The presentations seem to focus on selecting HEI as analytical partners. Can you please confirm if HEI is preferable?
Any organisation, so long as they have the necessary experience in real world evidence study design, conduct and analysis, can be the analytical partner. There is no preference as to organisation type.
The eligibility guidance for an analytical lead states that Commercial organisations including large enterprises and Small and Medium Enterprises (SMEs with a staff headcount no greater than 250 and an annual turnover no greater than €50 million, including start-up or spin-out companies) can apply. For large organisations is there a limit on headcount and turnover or does this only apply to SMEs? Is there a limit on turnover and headcount for subcontractors?
Any size commercial organisation can apply or be a subcontractor. There is no limit on headcount, this is only included in the guidance to support the definition of a small or medium-sized enterprise.
Can there be two analytical partners - i.e. a HEI and a Health Innovation Accelerator working in partnership? Is this encouraged?
The application should be lead by one analytical partner with experience in real world evidence study design, conduct and analysis. Additional expertise may be provided by other collaborators where required and justified.
Do analytical partners already need to have a technology partner to submit, or can they submit as separate entities to form a constortia selected by NIHR?
The application must include all consortia members - an analytical partner, technology partner(s) and site(s)/service(s); NIHR is not putting the consortia together. If you are an analytical or technology partner looking for an analytical/technology partner, you can register for the matchmaking event or contact the support listed in the Supporting information on the website.
Will analytical partners be eligible to lead more than one consortium? Will there be a limit on the number of applications an entity can collaborate on for this call?
There is no limit on the number of applications an analytical partner may submit or a partner may collaborate on, however, it is recommended that lead applicants focus on their strongest application. If submitting multiple applications, you should clearly indicate the differences, and where applicable, potential synergies. You would have to ensure you had resource to complete all activities if funded. Overlap for activities and costs would not be allowed.
Can a technology partner appear in more than one bid? For example if multiple NHS regions/HIN's want to do a study on our technology.
There is no limit on the number of applications a technology partner may collaborate on. However, you would need to ensure you had sufficient resource to complete all projects, if successful. If you are involved in multiple applications, you should justify the strategy for multiple projects (rather than one project with multiple sites,) and clearly indicate the differences, and where applicable, potential synergies. Overlap for activities and costs would not be allowed.
We are a mental health trust and would would be very open to collaborate, do we have to complete a partnering form too?
You can contact the eligible technologies from the list directly to initiate collaboration discussions. You may also send your details to i4i-rwe@nihr.ac.uk with confirmation that you agree to your details being shared with potential analytical and technology partners. This list will be available to attendees at the matchmaking event or on request from potential analytical or technology partners.
Does NIHR allow for NHS sites to use two products on the EVA? We have been approached to have our product tested against another EVA
The aim of the call is to generate real world evidence to support full NICE evaluation and widespread adoption. There is no restriction on the number of technologies to be deployed at a site, and any number of technologies under one EVA topic can be included on one application, so long as there is resource to complete all proposed activities if multiple applications are funded.
Is it possible to have several adoption sites (any limit?)
There is no limit to the number of sites you may include in your study. The overall strategy and design of your project should be outlined within your application, as well as how the chosen locations and types of settings are suited to address the evidence needs of the product(s). A maximum of 15 co-applicants can be included in each application. If you have more sites than you can include, please detail the extras in the 'Detailed Project Plan' section.
I presume this call is open to organisations in Scotland (some of the previous slides only showed English and Welsh networks)?
Yes, all partners, including adopting sites, can be based anywhere in the UK
Can partners in the consortia come from outside of the UK?
Proposals should be led by an analytical partner that is a UK legal entity partnered with UK sites/services. Technology partners should be those that have been recommended for early use in the NHS through NICE EVA. Other collaborators or sub-contractors may be from outside the UK if suitable justification is provided.
Can you submit applications that covers multiple EVAs on one theme (ie 4 mental health EVAs) or would this need to be 4 separate applications
Yes, you may submit one application containing any number of eligible technologies under one NICE EVA topic.
There are a number of EVAs in consultation that are likely to be published in the very near future - will they be included if published before the closing date for applications or not? Will there be another round of funding?
Only the technologies included on the list are eligible for this call. However, you will notice a "status update" column showing Draft or Published. If the draft guidance of the technologies on the list is published by the close date, they are included and can apply. The i4i NIHR & OLS RWE call is a one-off competition and at the moment further rounds have not been planned.
For the virtual ward technologies, the guidance seems to suggest that in addition to the 13 technologies listed, other technologies are potentially also eligible to apply. Can you confirm that this is the case please?
Although the Virtual ward platform technologies for acute respiratory infections assessment acknowledged the existence of multiple technologies, the virtual ward platform technologies eligible for the NIHR i4i & OLS RWE Call are only the 13 technologies (included on the list of eligible technologies) where developers participated in the NICE evaluation and provided information to NICE and have as such been reviewed and recommended for early use in the NHS whilst further evidence is generated.
The virtual ward platform technologies EVA is focused specifically on acute respiratory infection but evidence gaps exist for other pathways. Could the submission be expanded to include other conditions?
One of the main aims of this funding call is to generate the evidence for the full NICE evaluation and those required by the commissioners for the widespread adoption. While there may be gaps for the evidence of virtual wards in other pathways, acute respiratory infection is the use case evaluated by NICE EVA and recommended for further evidence generation. Therefore, the projects should include only activities related to the specific use case recommended by NICE.
Would you be able to confirm what conditions you would class as being covered by the Virtual Wards for acute respiratory infections EVA? For instance, would addressing COPD exacerbations fall under this?
The NICE Early Value Assessment guidance on virtual ward platform technologies is for acute respiratory infections. COPD exacerbations where caused by acute respiratory infections could be considered a relevant subpopulation, but is not specifically mentioned in the guidance or evidence generation plan.
We are interested in looking at Virtual Wards for acute respiratory infections in children, will that be permitted?
Please check the scope of the conditional recommendations. For example, in this case they apply to people over 16 with acute respiratory infection
We have programmes listed in the NICE EVA for adults and in the one for children and young people. Is it possible to submit 2 applications, one for each population, since they are too distinct to be submitted under one. If so, would it be detrimental to have 2 applications or would the 2 have the same chances of getting funded?
Only technologies included under the same EVA topic can be included in the same application to this funding competition. If the technologies are under two different EVA topics then two separate applications would be required. Each application will be reviewed against the assessment criteria and awarded based on its own merit. However, if considering partnering on more than one application, you would need to ensure you had sufficient resource to complete all projects, if successful. You should also clearly indicate the differences, and where applicable, potential synergies. Overlap for activities and costs would not be allowed.
Within our application for the children and young people programmes, would it be possible to include our parenting suite of programmes to be considered within the evaluation framework of the project as we anticipate it would be leveraging the same infrastructure as the adolescent programmes?
Please check the scope of the conditional recommendations. For example, in this case they apply to self-help digital cognitive behavioural therapy (CBT) technologies used as an initial treatment option for children and young people (aged 5 to 18) with mild to moderate symptoms of anxiety or low mood.
Question related to the digital weight management technologies to support weight-management medicine EVA – the cost of weight-management medications (GLP1RAs etc) is substantial – did I understand correctly that drug costs cannot be included in the costing for this call and that funding for this must be sourced elsewhere?
The weight management medicine is commissioned, and the technologies themselves are used to support the prescription and monitoring of these treatments. Therefore, since the impact of interest is from the technologies supporting the medicines rather than the medicines themselves, companies would not be expected to have to cover this cost.
Why is HTE12 Artificial intelligence-derived software to analyse chest X-rays for suspected lung cancer in primary care referrals: early value assessment, not included on the list of eligble EVA technologies?
Only technologies that have been approved through EVA for early use in the NHS are included and those that have research-only recommendation (such as HTE12) are not within the scope of this call
We are already virtual ward tech in clinical practice in the NHS. Are we already too mature? Is this call more for more nascent technology?
This Call is for real world evidence generation activities to support full NICE guidance, as well as real world evidence to encourage increased uptake and use of technologies within the NHS, and to capture generalisable learnings. If you are an eligible technology and wish to carry out such activities (see supporting information for further information) then you are welcome to form a consortia and submit an application.
Should the site be at a pre-adoption stage, or would it be suitable for site to carry out post-adoption evaluation? E.g. if a site is currently trialling a tech, can we use this opportunity to collect data and perform evaluation?
If the data collected post-adoption can address the evidence gap then there is no reason why post-adoption data cannot be used. However, in some cases additional data may be needed, for example, a suitable comparison population receiving standard care alone. Funding is available to support evidence generation to address NICE EVA recommendations and also to support widespread adoption for eligible technologies.
Can a company be completing one NIHR grant and start this one?
All applications will be assessed for overlap by the NIHR team, therefore, ensure that there are no funding or project overlap between existing awards and your application for this competition.
Finance
Whilst you note no upper limit on budget, with 13 eligible EVA topics, it would be helpful to have some guidance on total budget pot available for this call. Are you able to provide an anticipated budget per project?
The Real World Evidence Programme is backed by £10m government funding. Project budget will vary by activities proposed as well as number of technologies and sites/services included so we do not have a specific guide but all costs should be justified.
Is there funding and ambiton to short list all projects for all the technologies listed in the EVA? if you are not short listed is there any opportunity to reapply?
The Real World Evidence Programme is backed by £10m funding. This is not enough to support all eligible technologies. Proposals will be reviewed against the assessment criteria (included in the Supporting information) and recommended for funding based on individual merit. This is a one-off pilot call but other funding mechanisms exist to support real world evidence generation, such as NIHR i4i, SBRI Healthcare and NHS Cancer Programme.
Is this competition structured as a grant? I.e. does the grant go to the company and then it is on the company to run their evidence generation program and distribute directly to trusts? Or do we go through a procurement channel where the technology can then be procured directly by trusts?
The funding from this Call is an award. Funding will be disbursed to the Lead Applicant (analytical partner) as outlined in the contract, subject to agreed milestones being met. The funds are then distributed by the analytical partner to the technology providers, sites and any other subcontractor as necessary. This is not a procurement process.
Does the funding cover the cost of the equipment/technology that we would use to genererate RWE
Cost of the technology to generate the real world evidence, but without any profit attached, would be eligible if justified (large organisations would be expected to provide the technology in kind). The supporting information and finace guidance provide further information for eligible costs.
In addition to costs of the technology, would any additional implementations costs be covered?
Implementation costs would be eligible if justified. The supporting information and finace guidance provide further information for eligible costs.
As a very small technology partner, what costs of the technology partner are acceptable in the application? Which of these might be included: technical support, software development, usability studies, product development, server rental, software licences, ... ? Are any of these reimbursed at 100% or are all costs reimbursed at 50%?
Cost of the technology to generate the real world evidence, but without any profit attached, would be eligible if justified (large organisations would be expected to provide the technology in kind). Costs of the technology partner to provide and support the technology during the project would be eligible, this can include technical support, software licenses, etc for the specific EVA-recommended product(s) used. Small changes to a technology that might be needed for its optimisation during the lifetime of the project, for example any adaptations deemed required for end user acceptance as part of the evaluation (but not requiring any further regulatory approvals) could also be considered. All eligible costs are reimbursed at 100%.
Will the NIHR fund treatment costs?
NIHR Awards do not include NHS support and/or treatment costs. These costs, including costs for Social Care research, are funded via Clinical Research Networks (CRNs) and should be detailed in the Schedule of Events Cost Attribution Tool (SoECAT). Further support with treatment costs can be provided by your local CRN.
Can salaries be covered for technology or adopting site partners, or only for analytical partners?
Salaries for the work performed by any applicant on the project can be requested.
NIHR Awards do not include NHS support and/or treatment costs. These costs, including costs for Social Care research, are funded via Clinical Research Networks (CRNs) and should be detailed in the Schedule of Events Cost Attribution Tool (SoECAT). Further support with treatment costs can be provided by your local CRN.
Innovators may apply for further funding from organisations such as NIHR, SBRI Healthcare and NHS Cancer. It is recommended to sign up to funding newsletters to be kept updated on future funding opportunities.
IP
Can I please ask how IP can be / should be managed in the project? If the project is a research study led by the evaluation partner, the lead partner (sponsor) will be owner of the arising IP and data; this is a recurrent problem in collaborative projects.
The NIHR's preferred position is that the contracted institution owns all Foreground IP resulting from the project. However, alternative ownership of Foreground IP may be necessary due to pre-existing arrangements or to simplify the route to patient benefit. The partners should determine the most appropriate ownership and management of IP and provide justification that this is the best route for achieving patient and public benefits.
Consortia
How can you sign up as an analytical partner?
There is no need to register separately to be an analytical partner. Analytical partners should be the lead applicant and have experience in real world evidence study design, conduct and analysis. See the full list of technologies which have been recommended for early use in the NHS through NICE EVA if you wish to make contact directly with a technology partner, or you may register to attend the matchmaking event.
Is it possible to bring more than one analytical partner so that there is a lead applicant partner and other academic partners taking on work packages related to other areas that area in scope for the project.
Each project needs to have one analytical partner as the lead applicant, however, you can include collaborators and academic partners to contribute to different work packages.
Is there any portal for information sharing/contact between the technologies and potential analytical partners/adopting sites prior to the matchmaking event? Presumably all 59 technologies are well aware of this fund?
Potential consortia members are welcome to contact potential partners at any time prior to call close. See the list of technologies which have been recommended for early use in the NHS through NICE EVA. Interested sites are welcome to send their details to i4i-rwe@nihr.ac.uk with confirmation that they agree to their details being shared with potential analytical and technology partners. This list will be available to attendees at the matchmaking event or on request from potential analytical or technology partners. All eligible technology partners have been contacted directly by NICE and made aware of this call.
Is there a list of qualified analytical partners for other eligible partners to refer to? Or are we waiting for those who have submitted EoI to NIHR / wait for matchmaking event?
NIHR does not have an exhaustive list of analytical partners and does not provide direct matching. Applicants are encouraged to take advantage of the matchmaking event and offerings by the Business Development team at NIHR to initiate discussions. They can also contact HINs, MICs, ARCs, HEIs or commercial companies directly as potential analytical partners.
Interested in hearing more about the process for potential 'match' between suppliers and analytical partners and the advice being provided to potential suppliers.
Partners will form consortia independently of NIHR and this Call, however, to support this process, we are holding a networking event for potential analytical partners and eligible technologies to initiate discussions.
If we already know what Analytical Parter we would like to use (we are a Technology Partner), then is the matchmaking event on Feb 13th not useful? Or is there some other match matching taking place there (e.g. with participating NHS trusts?)
The matchmaking event is primarily to support networking between potential analytical and technology partners who do not already have partners. However, there will also be talks from Innovators who have previously navigated late-stage product evaluations and NICE commenting on common issues and pain points.
What do I do if several companies want me to be a partner? How do I manage that?
It is up to partners to discuss options and select the best partners for the evidence generation requirements. There is no limit to the number of applications a partner may collaborate on, however, it is recommended that applicants focus on their strongest application. If submitting multiple applications, you should clearly justify the strategy, indicate the differences, and where applicable, potential synergies. You would have to ensure you had resource to complete all activities if funded. Overlap for activities and costs would not be allowed.
Do NHS adopting sites have to be co-applicants on the application? Are any NHS site costs covered – i.e. are sites expected to engage of their own resource? What NHS adopting sites can be covered?
Yes, NHS adopting sites, as one member of the consortia have to be co-applicants on the application. NHS research costs can be included in the application, while NHS treatment costs should be included in the SoECAT form. Please refer to the SoECAT guidance on our website and consult with your local CRN and/or AcoRD specialist.
Can you explain a bit more how can we leverage the health innovation network as a technology partner to find a relevant analytical partner? There are many different actors so how to best look into this and how to know there is indeed a relevant partner for us?
Support available to prospective applications is described in the Supporting Information, including a link to the Health Innovation Network (HIN). You may also want to sign up to the NHS Innovation Service to respond to your specific needs, including the most appropriate HIN.
What is the intended outcome if the expected patient and system benefits are met? Will the test sites have to commission the technology longer term?
Evidence collected during the project may support full NICE evaluation and/or adoption. There is no obligation for sites to continue using the technology following the completion of the project. This should be discussed amongst partners themselves.
Application
Will any applications be reviewed prior to the deadline?
No, all reviews will take place after call close (1pm 27 March 2024)
What is your definition of RWE?
We use the NICE definition provided in their real-world evidence framework.
Can the evaluation be shorter than 3 years? Some technology may not need the 3 year windown and can effectively generate the evidence in less than 1 year.
Proposals should only be for the time required and up to a maximum of 3 years. The aim is to accelerate the adoption of innovative technologies resulting in faster benefits for patients and service-users.
The NICE EVA evidence generation plan for weight management technologies recommend study design that takes 4 years, but the funding call is just for 3 years. Can you give any guidance on how to tackle this?
Most of the evidence generation would occur within 3 years, even using the outlined before and after design. However, prospective cohort design may allow for evidence generation within 2 years of follow up, provided a suitable comparison service can be found.
Are there any early value assessments for any ongoing MedTech researches that we can study please?
Published and draft Early Value Assessments are available on the NICE EVA website
Is there an evidence plan for all technologies that have been through EVA, which set out suggested approach/study design?
Evidence Generation Plans are published on the NICE EVA website for all technologies recommended for early use in the NHS.
Some EVAs may have been developing evidence since EVA publication. What is the process in place to help determine whether this evidence is 'good enough' to reduce the need to generate the evidence initially recommended?
Please include a description of evidence generated to date and the outstanding evidence requirements for adoption within your application so this can be reviewed alongside your proposed project and the Evidence Generation Plan.
Would the panel accept proposals that ‘go beyond’ NICE, but offer good value for money for the data collected, or is it better to stick to the NICE evidence generation plan exactly, potentially wasting the opportunity to get extra data?
Yes, the Committee would accept plans to collect data in addition to that in the NICE evidence generation plan to help with future NHS adoption and roll-out
Please can you expand on the non evidence generation requirements please. For commissioners, for example.
In addition to the evidence that is generated to address the gaps identified by NICE evidence generation plan, we would expect the applications to include generation of evidence that would be needed by the comissioners to help increase the uptake of the technologies in the NHS. There may be evidence that is required by commissioners that are in addition to those requested by NICE, and we would expect that a strong application to consider both.
Could you please expand on the generalised knowledge dissemination requirement? Would you see this as completely separate to an implementation process evaluation (or perhaps an augmentation of process evaluation?)
One of the key aims of this award is to derive generalisable knowledge on real-world evidence and implementation to support future innovators to improve the clarity and quality of real-world evidence and enhance understanding of what commissioners want to see to encourage increased uptake. We would expect a work package within the project plan with clear deliverables to outline how you will capture your learnings during the project and NIHR/OLS may hold discussions with successful applicants to bring things together and ensure appropriate dissemination
As the focus is on real-world evidence, can I check that if for pratcical and value-for-money reasons, a pragmatic RCT can be proposed? Also, can two study designs be proposed if there are two settings of interest to the EVA for the same/similar intervention?
Yes, the Committee would accept plans to collect data in addition to that in the NICE evidence generation plan to help with future NHS adoption and roll-out. Furthermore, differences in the study design or analyses may also be acceptable where these can be justified and are likely to improve the robustness of evidence.
Should evidence generation be done as HRA approved research or service evaluation?
NIHR would not be able to advise on the study design and this should be done by the analytical partenrs and collaborators experienced in RWE design and by consulting the evidence generation plan. However, please note that the proposed study design will be assessed by the committee.
Is it a requirement to work with a Secure Data Environment (SDE)?
It is not a requirement to work with an SDE but support to navigate the SDEs and the data within them will be available to successful applicants
What patient and public representatives would NIHR suggest to inform the proposals and projects?
We would suggest you look at NIHR PPIE resources for applicants. You can also approach charities and ues the People in Research website.
Is the process pass or fail at the first stage of submission or is there opporunity for discussion and adaptation following submission and initial assessment?
This is a one-stage competition. Applications which are shortlisted will be invited to present to the Funding Committee followed by the opportunity to respond to the Committee's questions, but no further application submission is required.
Are you able to share details of the reporting expectations, if successful, and to whom.
NIHR will be managing the monitoring of successful projects. Quarterly reports and project meetings will be required. Engagement will also be required to complete the generalisable guidance required for this call.
Other
Are you working closely with NHS England ?
NHS England colleagues are on the working group involved in this call. Additionally, Data for R&D Team within NHS England’s Transformation Directorate will be able to provide signposting to relevant NHS data and organisations who will be able to support awardees in generating evidence.
Is there any connection with IDAP Innovative Devices Access Pathway?
Whilst the IDAP and the Real-World Evidence Programme are not directly connected, they are both part of a suite of interventions stemming from the Life Sciences Vision aiming to create an environment in which patients and the NHS can receive a real benefit from new innovations and industry can grow and succeed in the UK
Can other sites trial the technology in tandem with this EVA evidence generation - or are they expected to wait until this trial completes? Will this potentially delay the technology scale up?
NIHR does not place any restrictions on the use of the technology during this competition.