Preface note: I’ll use the abbreviations for Real World Data (RWD)  and Real World Evidence (RWE) in this article. Other abbreviations will be defined the first time they appear.

What are the challenges in Using RWD/RWE? I recognized there would be practical issues in collecting and using RWD as well as consent and privacy issues. However, I was surprised by how many of these issues have been thoughtfully addressed by scientists and policymakers. In this article, I explain several of the challenges and provide some useful references if you want to know more.

In the introduction to my June article, “FDA Guidance for Real-World Data and Evidence,” I said, “This increased activity raises questions about practical approaches to collecting and using RWD in generating RWE and concerns about patient data privacy, ownership, and informed consent.” 

This statement was the springboard for this article. I thought the focus would be on privacy and ownership. However, more practical considerations of acquiring, cleaning, sorting, and analyzing RWD present greater challenges right now. In this article, I’ll review some key points about how RWD/RWE differs from “normal” clinical trial data, some of the technical issues specific to the field, and how privacy fits into the picture. I’ll also share some key changes needed to make RWD access and conversion to RWE feasible for the long haul. I’ll end with an FDA RWD/RWE update since my post in June. 

It’s important to remind ourselves that RWD is different than data from randomized controlled trials (RCT) data. RCTs tend to be explanatory in nature, with a focus on the efficacy of the intervention under ideal conditions. While RCT are required for regulatory approval of novel medical products, the studied population is highly selected and monitored closely for adherence to the protocol. The measurements acquired during an RCT (e.g., lab tests or clinical actions) are indirectly related to the desired health outcomes, with limited utility for a clinician, patient, and policymaking decisions. RWD is gathered in the context of pragmatic trials, which focus on the question, “Does this intervention work in normal practice?” The investigators are more likely to be routine practitioners, and the inclusion criteria less restrictive, targeting a population that is more representative of a typical clinical setting. Parameters that would complicate an RCT, like the healthcare delivery system and patient-related factors, are more easily studied in a pragmatic trial. Contrary to common belief, RWD/RWE is also compatible with randomization of treatments. The use of RWD/RWE make pragmatic clinical trials more informative for patients and their caregivers, too. The reference for this section summarizes how explanatory and pragmatic trials are related and provides a practical, easy to understand, example using the VERVE trial. I recommend it to readers interested in learning more about pragmatic trials.[1] 

Where do RWD come from? The FDA defines the potential sources as:

1. Electronic Health Records (EHRs) – these records, from primary, urgent, and home care visits, contain structured data like test results and diagnostic codes or less standardized data like clinical observations or patient intake sheets.

2. Administrative claims – insurance claims from healthcare service providers. In the US, this is typically claimed under the Medicare and Medicaid programs.

3. Patient-Reported Outcomes (PROs) – reports of health status that come directly from the patient without the mediation of a care provider.

4. Patient-Generated Health Data (PGHD) – data from devices like mobile applications or home health devices (like blood pressure monitors).

5. Medical product/device registries – created after approval, these registries collect data to enable the postmarketing surveillance required by most regulatory agencies.

6. Condition or disease-specific registries – containing condition-specific information from patients, usually in more detail than in the EHR.

7. Environmental factors and social determinants of health – data collected using direct patient surveys or extracted from community-oriented resources. 

PRO and PGHD are both primary data sources. The other five sources are secondary, typically collected for purposes other than research. This fact has implications for a source’s fitness and capacity to answer a research question, as well as its patient permission requirements and ease of access. Teams at the NIH have been studying RWD/RWE as part of a broader effort to develop designs and tools for the conduct of pragmatic trials. They have assembled the current best practices into an online guide for researchers and the public. The Data, Tools, & Conduct section covers RWD/RWE in more detail. [2]

It seems obvious that patient concerns must be a key consideration in studies using RWD/RWE, given the notorious history of the Tuskegee syphilis study and other experiments on vulnerable populations. These infamous activities have led to the present-day protocol review infrastructure known as the Institutional Review Board (IRB) as well as informed consent requirements. The Common Rule (21CRR Parts 50 and 56) governs research involving human subjects conducted in the US, and recent changes to this rule have made it easier to reuse data collected during routine healthcare operations. However, IRB approval of the study protocol remains prudent for pragmatic trials using patient data. Health Insurance Portability and Accountability (HIPAA) regulations require protected health information (PHI) only be released under certain controlled conditions. Datasets from healthcare organizations that are not de-identified (they still contain some PHI) require additional PHI removal or special data usage agreements before research use. Data from government agencies often have similar restrictions.[3] 

Patient acceptance of RWD/RWE use and the accessibility of trial outcomes are another area of concern. Engagement is the subject of considerable effort by the National Health Council (NHC). They have convened several forums to hear patient concerns. The first, held in 2017, had four key findings:[4]

•  Patients recognize how they might benefit from the use of RWD to understand treatments in diverse populations. 
•  Education efforts for patients are necessary so they can better use and contribute to RWE.
•  Clinicians are more familiar with randomized controlled trial data, but they must understand and accept RWE to achieve patient benefit goals.
•  Social media sharing of patient experiences can generate RWD, especially for rare conditions. However, decision support tools are needed to distinguish between good quality RWE and “junk science.

These findings have informed subsequent NHC-sponsored forums and FDA efforts to incorporate RWD/RWE in regulatory decisions. Their RWE Classroom, created in partnership with the Duke-Margolis Center for Health Policy, contains links to the forum reports, a training series on RWD/RWE, a glossary, and explanations of key topics. 

Another challenge is that much of RWD currently in EHRs is not easily usable or searchable because it’s messy, in difficult-to-use formats, or uses nonstandard terminologies. Pharma companies are familiar with these challenges. Making sure clinical trial data is clean in content, canonical in format, and consistent with the master library of terms occupies a huge number of person-hours. Translating this concept into the process of collecting RWD will require thoughtful engagement. The systemic changes needed to enable greater use of RWD/RWE in pragmatic trials are summarized in an article by Aracelis Torres for the VeranaHealth blog.[5] She notes these best practices are for healthcare organizations, but I believe they won’t be effectively implemented without a public-private collaboration of vendors, providers, policymakers, and standards organizations. I’ve summarized her seven points into four categories:

Standardized data formats for EHRs – I’ve posted about this previously, but it can’t be repeated too often or loudly. Instead of proprietary formats, EHR vendors should convert to internationally recognized standards such as Systematized Nomenclature of Medicine Clinical Terms (SNOMED-CT) and International Classification of Diseases (ICD). This is the top change that must occur, on which other changes will depend for their highest value-add. Using these standards (already integrated into the data collection and analyses protocols for RCTs) will facilitate knowledge transfer between RWD/RWE and RCT data scientists and advance the translation of medical data into health knowledge and practice. 

Easier data collection systems that facilitate quality – If you’ve been to the doctor in the past few years, you’ll recognize the effort needed to encode your visit into your provider’s EHR system. Designing novel incentive systems to improve data entry and adding natural language processing and AI tools to facilitate text mining and data cleaning/verification will reduce the burden on healthcare operations teams and increase the data’s content value.

Innovate new data platforms to simplify analytics – This is a natural extension of adopting standardized data formats. Making sense of large data sets is a challenge common to multiple industries. Feed-forward application of AI and ML tools and best practices from diverse fields like e-commerce and finance to transform RWD into RWE is a valid target for digital health entrepreneurs. The platform outcomes may spark complementary innovation into the founding sectors.

Serve the patient community first and foremost – Without transparency, engagement, and meaningful data privacy, RWD initiatives will fail to meet their promise and poison the well for future growth. Similarly, modern ethics require reducing bias and increasing equity, which means the RWD collected must represent the true diversity of the populations.

The outcome of implementing these best practices will be to advance and sustain clinical research and human health goals through the rest of the century. 

 The FDA has issued one guidance and a new program since my June article. “Patient-Focused Drug Development: Selecting, Developing, or Modifying Fit-for-Purpose Clinical Outcome Assessments” describes how experience data and other relevant information from patients and caregivers may be collected and used for medical product development and regulatory submissions. It’s the third in the four-part Patient-Focused Drug Development (PFDD) guidance documents. “Externally-led Patient-Focused Drug Development Meetings” springs from the agency recognizing that there are many more conditions than are included on the FDA’s current PFDD meeting list. They have established a program to help patient organizations plan for and engage the agency with an externally-led PFDD meeting. The website contains an overview of the program, guidance on planning an EL-PFDD meeting, a list of upcoming planned meetings, and the agency’s full list of Condition-Specific meeting reports.

My research following the completion of my June article showed me that, while the promise of RWD/RWE for clinical research is alluring, there remain multiple challenges to execution. I’ve shared some of those challenges in this article. Fortunately, serious study has been underway to address these points. Readers interested in learning more are encouraged to review the references and the NHC’s RWE Classroom online. We can anticipate there will be meaningful therapeutic advances to come from the use of RWD/RWE.

References

[1] – Curtis, J. R., Foster, P. J., & Saag, K. G. (2019). Tools and Methods for Real-World Evidence Generation: Pragmatic Trials, Electronic Consent, and Data Linkages. Rheum Dis Clin North Am, 45(2), 275-289.

[2] – NIH Pragmatic Trials Collaboratory, Rethinking Clinical Trials®. Retrieved from https://rethinkingclinicaltrials.org/table-of-contents/

[3] –  Ibid., Acquiring Real-World Data, Section 6, Gaining Permission to Use Real-World Data.

[4] – Oehrlein, E. M., Graff, J. S., Harris, J., & Perfetto, E. M. (2019). Patient-Community Perspectives on Real-World Evidence: Enhancing Engagement, Understanding, and Trust. The Patient – Patient-Centered Outcomes Research, 12(4), 375-381.

[5] – Torres, A. (2022). Best Practices for Transforming Real-World Data into a Quality Asset for Healthcare. Retrieved from https://www.veranahealth.com/blog/2022/02/best-practices-for-transforming-real-world-data-into-a-quality-asset-for-healthcare/.