Harnessing Real-World Data To Improve Assessment Of Health Interventions
Randomized clinical trials (RCT) are generally considered the gold standard to establish a treatment effect. However, RCTs have limitations that often compromise generalization of results such as: small sample size, strict exclusion criteria, short follow-up duration, and poor representativity of social determinants of health (Togo & Yonemoto, 2022). Provided that the data fulfills appropriate requirements, real-world data (RWD) have become a source of complementary information to RCTs that can fill research gaps and better assess treatment effectiveness, especially in subpopulations (Schad & Thronicke, 2022). More specifically, RWD can provide information on natural history and course of disease, multiple outcomes, safety surveillance, and the investigation of treatment patterns in real-world practice (M. Liu et al., 2022). This white paper explores three key factors relevant to evidence generation that are accelerating the use of RWD to improve assessment of health interventions: 1) the growth of RWD and evidence from RWD (RWE); 2) advances in research-design and data analysis; and 3) new legislation and guidelines regarding regulatory science.
1) The Growth of RWD and RWE
- Sources of RWD
Increased digitalization of RWD has revolutionized the way we understand and approach healthcare, research, and decision-making. RWD is the data related to health status and healthcare acquired in routine clinical practice (US Food & Drug Administration, 2023a). The most common sources of RWD are: claims, electronic health records (EHRs), patient registries, and patient-generated outcomes (e.g. wearables) (Table 1). The importance of RWD has increased with expansion in the use of EHR, the increased proliferation of consumer digital technologies, and, the improved means of capturing, storing, and analyzing longitudinal data of patients. As RWD has evolved, its potential to drive advancements in healthcare and improve outcomes has become increasingly evident.
There are many instances during the course of a patient’s treatment where RWD is being passively collected. For example, consider the case of an individual who had a neurological injury. Upon admission to the healthcare facility, the patient’s visit will be documented in an EHR managed by the healthcare team. The EHR contains patient demographics, diagnosis, medical past and current history, clinical assessments, procedures, prescribed drugs, laboratory results, progress notes, supporting diagnostic images, and admission/discharge notes. During the rehabilitation, advanced technologies can contribute with assessments and monitoring at a smaller level of granularity compared to standard clinical assessments (Scott & Dukelow, 2011)
For instance, upper limb robotics can track patient progress and record arm trajectory, range of motion, speed, level of assistance/resistance provided by the device, and level of difficulty of treatment. These insights can uncover impairments underlying functional recovery and provide objective data to personalize and adapt interventions, with its use potentialized when combined with other technologies. Finally, when the patient leaves the facility to the next level of care, medical insurance claims data is collected, which contains information on reimbursement for medical services from hospitals, clinics, and pharmacies. Although the original purpose of claims is for payment purposes, they have been used to estimate patient’s behavior, drug’s use and interactions, disease diagnosis, disease prevalence, and disease progression (F. Liu & Demosthenes, 2022). Connecting these diverse datasets makes it possible to individualize treatment decisions, improve outcomes, and to identify patterns at population level.
RWD can also go beyond the healthcare space and include broader datasets from health consumer technologies. Examples include mobile health apps (mHealth, e.g., Noom) and wearables (e.g., activity trackers: fitbit & apple watch). Wearable-based applications encompass many areas, including the assessment and monitoring of sleep, movement, metabolic, cardiac, respiratory, immunology and rheumatoid outcomes (Izmailova et al., 2018). Wearables have also been incorporated in clinical trials for remote and real-time monitoring during drug development (Swift et al., 2018). An interesting case monitored ambulatory heart rate for 14 days using an adhesive patch with a single-lead ECG. This protocol had 57% better detection rate for diagnosis than the “gold standard” 24h Holter monitoring (Izmailova et al., 2018). In patients with stroke, wearables were used to identify that gains in motor performance traditionally measured in clinics lack carry-over to usage in the real-world (Lang et al., 2023). These advancements hold great promise in developing new and more relevant endpoints and improving patient care and overall health outcomes.
Another type of RWD is the study of patient cohorts in patient registries. Patient registries are data repositories of groups of people defined by a particular disease/condition or use of a specific product. They are created to follow a subpopulation in the long-term to serve a pre-determined scientific, clinical and/or public health purpose. The main goals of patient registries are to enhance understanding of natural history of disease and uncover adverse events in understudied patient populations. In the U.S., there are two active stroke registries: the American Heart Association’s (AHA) performance improvement program called Get With The Guidelines-Stroke (GWTG-S) and the Centers for Disease Control and Prevention-funded Paul Coverdell National Acute Stroke Registry (American Heart Association, 2023; Centers for Disease Control and Prevention, 2023). These registries provide critical insight into clinical practice and disparities in healthcare delivery, allow for the surveillance of trends in treatment and quality of care, and are useful in evaluating guideline implementation and adherence.
- RWE
When RWD is implemented to provide clinical evidence about the usage and potential risks/benefits of a medical product, it is denominated RWE (US Food & Drug Administration, 2023). RWE can be utilized to inform regulatory decision-making to support the approval of new indication of drugs (label expansion), serve as confirmatory evidence for drugs approved through expedited regulatory programs, contribute to health technology assessments, and support assessment of the clinical and economic value of medical interventions (Swift et al., 2018). As an example of a RWE use case in robotics, in March of 2023, FDA approved the use of an exoskeleton for individuals with spinal cord injury in stairs and curbs in the US (Exoskeleton Report, 2023). This exoskeleton had already been approved to be used solely in handicap-accessible structures in healthcare facilities since 2011, and for personal use since 2014. However, the delay in approving its application in other surfaces was due to a lack of data, so the submission leveraged RWE from a prior opening in 47 countries in the European market to extend its use.
RWE has been employed for regulatory submission, as well as for other activities during clinical development in the pre-approval and post-launch phases of drugs and medical devices (Togo & Yonemoto, 2022). Pre-approval trial designs can provide valuable information on the natural history and progression of a disease, standard of care practices, and help identify areas of high unmet medical needs. On the other hand, post-approval studies are crucial in evaluating the actual utilization, treatment patterns, comparative effectiveness, and safety of medical products. These studies are essential in demonstrating the overall value of the product and healthcare providers on important therapeutic findings that can guide real-world treatment decisions. In addition to the RWE, RWD applications go far beyond regulatory applications and clinical development and can support other disciplines such as Health Economics and Outcomes Research and Populational Health.
2) Advances in research-design and data analytics
2.1) Research-design
A rigorous methodological consideration is critical to validate the evidence and to make best use of RWE for regulatory purposes and in healthcare decision making. As in any research, the first step of RWE is to define a research question. According to Gokhale et al. (2020), having the thought process of a research study forces investigators to design the intervention, comparators, timelines, outcomes, and confounders. The use of guidelines for data reporting is encouraged to increase transparency (please see these references: Strengthening the Reporting of Observational Studies in Epidemiology (STROBE, 2023) and Reporting of studies Conducted using Observational Routinely-collected Data (RECORD, 2023)). Some of the key points for protocol development of RWE studies are:
(1) Clearly articulated research question;
(2) Fit-for purpose data source;
(3) State-of-the-art design with proper control variables and comparators;
(4) Analytic methods, including sensitivity analysis;
(5) Likelihood of being able to reasonably replicate a study in another similar setting.
A common misconception is that there exists a dichotomy between RWD and RCT and that RWD consist of only observational and non-interventional. Although treatment randomization is a key strength of RCT, not all clinical trials are randomized and this depends on the type of protocol developed for the research question. The degree of reliance on RWD varies with the type of study design (Fig. 1) (Concato & Corrigan-Curay, 2022). The strongest counterexample to this is the pragmatic clinical trial. Pragmatic clinical trials are interventional studies recommended when the treatment effect on a heterogeneous population is needed, the optimal treatment is largely unknown in everyday practice, or medical needs are insufficiently met (Liu et al., 2022). While this type of design can be challenging to operationalize, there are advantages to using a more pragmatic design in cases where patients’ inclusion/exclusion criteria may be important in determining the efficacy of specific treatments. This is often the case in trials testing advanced technologies to improve functional outcomes from neurological conditions at inpatient facilities. These patients may have many co-morbidities and, as such, a large portion of them would be considered for treatment using the devices in routine care with close monitoring of symptoms by therapists. For instance, those with orthostatic hypotension, a common condition after long periods in bed and which can be successfully managed by slow transition to sitting and upright postures are typically excluded from an RCT that needs to comply with strict inclusion and exclusion criteria. Another important class of co-morbidities that are treated in routine practice, yet excluded from RCTs, are psychosocial issues (substance abuse, depression, anxiety, PTSD). Pragmatic trials are the strongest type of design because randomization addresses threats to internal validity, while RWD addresses threats to generalizability.
Most study designs considered for regulatory decision-making are hybrid and they combine RWD and clinical trials (Doyle, 2019). There are three main types: single-arm, extension, and enrichment. Typically, the single-arm trial is when a sample of individuals with the targeted medical condition is given the experimental therapy and then followed over time to observe their response. This type of single-arm design is useful in the case of rare diseases when randomization may not be feasible or ethical. Additionally, single-arm trials can be associated with external controls of a previous trial, aka “augmentation”, “synthetic patient cohorts” or “virtual control arms”. In the extension trials, data from RCTs is linked to innovative methodologies that monitor long‐term outcomes intended for post-marketing requirements of safety. The capture of RWD can be expanded through methodologies such as decentralization trials (e.g. trained nurses), direct‐to‐patient approaches (e.g. wearables, patient‐reported outcomes), or databases (e.g. registries, claims) (Andre et al., 2020). Finally, in the enrichment trials, RCTs are linked to secondary real-world data (e.g., registries) to enrich the information. The importance of these RWE options for industry is that, when well conducted, they can be cost-effective and help bring treatments to market faster.
2.2) Data analysis
In traditional RCTs, the controlled nature offers an advantage in evidence generation as there are methods for reduction of systematic errors and biases (like randomization and double blinding) (Sherman et al., 2018). While details about the statistical methods for causal inference will not be covered here, some important aspects that can increase the robustness of RWE are adjustments for differences in baseline profile using statistical tools (e.g. the propensity score matching to create comparable cohorts in the real-world from retrospective treatment assignment), large samples, prospective designs, and a long and extensive follow-up (Liu & Demosthenes, 2022). Additional consideration related to methods for minimizing bias by study design and analysis can be found in Gokhale et al. (2020) and Togo & Yonemoto (2022).
3) New Legislation and Guidelines Regarding Regulatory Science
In 2016, the FDA approved the 21st century Cures Act. The “Cures” was designed to help accelerate medical product development and bring new innovations and advances to patients who need them faster and more efficiently. This act had many goals, one of which was the expansion in the types of study designs used for the evaluation of new treatments making it possible to use not only RCT data, but also RWE, to establish treatment effects (Fig. 2). In the years following the Cures Act, several additional guidelines and standards that clarify the use of RWE for regulatory decision-making for medical devices and drug approval were disseminated. Below is a list with the main developments in RWE for regulatory decision-making in the U.S. (Table 2). Also, the U.S. Food & Drug Administration compiled a document with 90 examples of RWE used in medical device regulatory decisions leveraging registries, claims, EHR, and other sources (US Food & Drug Administration, 2023b).
- Final considerations
The adoption of RWE into the typical clinical trial pipeline has the potential to make results from RCTs more applicable to routine clinical practice and drastically broaden the spectrum of patients that benefit from treatments. RWD and RWE can also identify and minimize health disparities based on gender, race, and ethnicity; bring treatments to market faster at a lower cost; and provide a more comprehensive picture of the adverse effects of interventions post market-approval, especially in those living in remote areas. As such, the legitimation of the use of RWE by regulatory authorities represents an important change in the landscape of data sources available to clinical researchers. Now, hybrid designs and pragmatic clinical trials can complement RCTs and/or enhance their external validity. However, the early establishment of cross-disciplinary teams and their engagement with regulatory authorities before conducting studies are crucial to promote wider acceptance of RWE in decision-making. To unlock the full scientific potential with the necessary rigor and standard, various stakeholders in the healthcare industry, government, patient populations, and research are working to support innovative study designs, address privacy concerns, and continue to improve the laws and guidelines governing regulatory approval of drugs and medical devices.
References
American Heart Association. Get With The Guidelines Stroke. Updated 2023. https://www.heart.org/en/professional/quality-improvement/get-with-the-guidelines/get-with-the-guidelines-stroke. Accessed 3 Jul 2023.
Andre, E. B., Reynolds, R., Caubel, P., Azoulay, L., & Dreyer, N. A. (2020). Trial designs using real-world data: The changing landscape of the regulatory approval process. In Pharmacoepidemiology and Drug Safety (Vol. 29, Issue 10, pp. 1201–1212). John Wiley and Sons Ltd. https://doi.org/10.1002/pds.4932
Center for Disease Control and Prevention. Paul Coverdell National Acute Stroke. Updated 5 April 2023. https://www.cdc.gov/dhdsp/programs/stroke_registry.htm. Accessed 3 Jul 2023.
Concato, J., & Corrigan-Curay, J. (2022). Real-World Evidence — Where Are We Now? New England Journal of Medicine, 386(18), 1680–1682. https://doi.org/10.1056/nejmp2200089
Evans, R. S. (2016). Electronic Health Records: Then, Now, and in the Future. Yearbook of Medical Informatics, S48–S61. https://doi.org/10.15265/IYS-2016-s006
Gokhale, M., Stürmer, T., & Buse, J. B. (2020). Real-world evidence: the devil is in the detail. In Diabetologia (Vol. 63, Issue 9, pp. 1694–1705). Springer. https://doi.org/10.1007/s00125-020-05217-1
Izmailova, E. S., Wagner, J. A., & Perakslis, E. D. (2018). Wearable Devices in Clinical Trials: Hype and Hypothesis. Clinical Pharmacology and Therapeutics, 104(1), 42–52. https://doi.org/10.1002/cpt.966
Lang, C. E., Holleran, C. L., Strube, M. J., Ellis, T. D., Newman, C. A., Fahey, M., DeAngelis, T. R., Nordahl, T. J., Reisman, D. S., Earhart, G. M., Lohse, K. R., & Bland, M. D. (2023). Improvement in the Capacity for Activity Versus Improvement in Performance of Activity in Daily Life During Outpatient Rehabilitation. Journal of Neurologic Physical Therapy, 47(1), 16–25. https://doi.org/10.1097/NPT.0000000000000413
Liu, F., & Demosthenes, P. (2022). Real-world data: a brief review of the methods, applications, challenges and opportunities. In BMC Medical Research Methodology (Vol. 22, Issue 1). BioMed Central Ltd. https://doi.org/10.1186/s12874-022-01768-6
Liu, M., Qi, Y., Wang, W., & Sun, X. (2022). Toward a better understanding about real-world evidence. European Journal of Hospital Pharmacy, 29(1), 8–11. https://doi.org/10.1136/ejhpharm-2021-003081
Nabhan, C., Klink, A., & Prasad, V. (2019). Real-world Evidence – What Does It Really Mean? In JAMA Oncology (Vol. 5, Issue 6, pp. 781–783). American Medical Association. https://doi.org/10.1001/jamaoncol.2019.0450
RECORD. Updated 2023. https://www.record-statement.org/. Accessed 3 Jul 2023.
Schad, F., & Thronicke, A. (2022). Real-World Evidence—Current Developments and Perspectives. In International Journal of Environmental Research and Public Health (Vol. 19, Issue 16). MDPI. https://doi.org/10.3390/ijerph191610159
Scott, S. H., & Dukelow, S. P. (2011). Potential of robots as next-generation technology for clinical assessment of neurological disorders and upper-limb therapy. Journal of Rehabilitation Research and Development, 48(4), 335–354. https://doi.org/10.1682/JRRD.2010.04.0057
Sherman, R. E., Anderson, S. A., Dal, G. J., Gray, G. W., Gross, T., Hunter, N. L., Lavange, L., Marinac-Dabic, D., Marks, P. W., Robb, M. A., Shuren, J., Temple, R., Woodcock, J., Yue, L. Q., & Califf, R. M. (2018). Real-World Evidence-What Is It and What Can It Tell Us? In The New England Journal of Medicine Downloaded from nejm.org at TUFTS UNIVERSITY on.
STROBE. Updated 2023. https://www.strobe-statement.org/. Accessed 3 Jul 2023.
Swift, B., Jain, L., White, C., Chandrasekaran, V., Bhandari, A., Hughes, D. A., & Jadhav, P. R. (2018). Innovation at the Intersection of Clinical Trials and Real-World Data Science to Advance Patient Care. In Clinical and Translational Science (Vol. 11, Issue 5, pp. 450–460). Blackwell Publishing Ltd. https://doi.org/10.1111/cts.12559
Togo, K., & Yonemoto, N. (2022). Real world data and data science in medical research: present and future. Japanese Journal of Statistics and Data Science, 5(2), 769–781. https://doi.org/10.1007/s42081-022-00156-0
US Food & Drug Administration. Real-World Evidence. Updated 5 Feb 2023. https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence. Accessed 3 Jul 2023.
US Food & Drug Administration. Examples of Real-World Evidence (RWE) Used in Medical Devices Regulatory Decisions. https://www.fda.gov/media/146258/download. Accessed 3 Jul 2023.
Daniela Mattos, Ph.D.
Scientist in Motor Control and Neurorehabilitation | Health Tech | RWD
I am a motor control and neurorehabilitation researcher with strong analytical skills, Physical Therapist (license from APTA), and medical devices consultant. I am passionate about using evidence-based approaches from traditional science and real-world data to improve peoples’ lives. I have 17+ years of experience in the domains of health technology (kinematics, EMG, finger and grip force, force plates), neurorehabilitation (stroke, amputees, hand transplant, intensive task oriented training, BCI), brain imaging (resting state fMRI and DTI), and medical devices (rehab robotics clinical applications and pressure sensor development). My scientific projects resulted in 25+ publications with over 643 citations, h-index of 13, 30+ professional presentations, and national and international collaborations.
