Overcoming data challenges in real-world evidence: Innovative approaches for enhanced data completeness and accuracy

Written by The Evidence Base
Editorials Events coverage Health data ISPOR Oncology and hematology Real-world evidence
Data completeness and accuracy

Maximizing data completeness and accuracy in real-world evidence (RWE) research is essential to ensure that research findings are valid, reliable, and reproducible. With fit-for-purpose data, stakeholders can draw greater insights, advance medical innovation, enhance regulatory processes, and make better informed decisions to improve patient outcomes.

At ISPOR 2024, Danielle Gentile, PhD and Richard “Scott” Swain, PhD, MPH of Cardinal Health explored innovative strategies towards maximizing data completeness and accuracy, sharing their innovative approaches towards real-world data (RWD) and physician-led chart collection in oncology. Here we take a Deep Dive into the session “Innovative Strategies for Fit-for-Purpose RWE Research: Maximizing Data Completeness and Accuracy“, summarizing the key takeaways and discussions from the presentation.

Common challenges in RWD

Dr Richard Swain began the session with an overview of the common challenges faced with using real-world datasets. These include accurately identifying rare patient populations, incomplete data, misclassification, and representativeness, which are all interlinked. For example, with incomplete or inaccurate data, it is difficult to account for differences between populations, making comparisons of safety and effectiveness between different treatment groups, or data sources, scientifically inappropriate.

Dr Swain outlined how these challenges can lead to bias, sharing an example of a Kaplan-Meier plot. The plot showed immediate separation between a Phase 2 trial arm and an external control arm. A visual inspection of the plot demonstrated various forms of bias, including immortal time bias, selection bias, differential loss-to-follow-up, and small sample size in the external arm.

A creative approach to RWD collection

To overcome these issues, Dr Swain went on to discuss an innovative approach to collect data for real-world research in oncology. Cardinal Health has developed provider- and practice-based research networks that collect deep patient-level clinical data via medical chart abstraction. The Oncology Provider Extended Network (OPEN) represents a large, geographically diverse network of community oncologists in the US. Through this network, Cardinal Health works directly with physicians to conduct physician-led chart reviews to collect complete and accurate data from patients that they personally treat, or from within their practice. Because OPEN physicians abstract information directly from the medical records they use and understand, Cardinal Health, and their clients, can gain insights into the reasons behind treatment decisions. This approach also enables advanced methods to measure safety and effectiveness outcomes with high precision. Similarly, with the Practice Research Network (PRN), a network of oncology practices, Cardinal Health works at the practice level to collect prospective, longitudinal patient data from chart reviews. The PRN is also capable of collecting patient reported outcomes, biospecimens, and of replicating some clinical trial methodologies, such as blinded independent central review.

Dr Swain explained that this approach offers several advantages over traditional data sources used in observational research; including:

  • Providing comprehensive and complete data
  • Featuring diverse, representative patient populations
  • Describing care settings in community oncology, where most patients receive treatment

This approach to RWD research ensures that the data accurately reflect real-world treatment patterns and outcomes.

Case study: Comparing demographic representativeness across RWE, clinical trial data and SEER-registry data

Dr Danielle Gentile shared a case study where Cardinal Health compared demographic representativeness across three data sources: RWD from OPEN, SEER-registry data, and clinical trial data [1].

Researchers at Cardinal Health identified OPEN studies internally conducted between 2017 to 2022 that corresponded with randomized controlled trials (RCTs) conducted during the same period and with similar inclusion/exclusion criteria, based on data available on clinicaltrials.gov. Data from the Surveillance, Epidemiology, and End Results Program (SEER) national cancer database were used as a proxy for a representative US patient population, accounting for the selection criteria used in the RWE studies and RCTs. Patient characteristics were assessed across the three data sources including by age, race, ethnicity, and sex assigned at birth (excluding breast cancer studies). The analysis spanned multiple oncology therapeutic areas, which included breast cancer, advanced renal cell carcinoma, liver cancer, non-small cell lung cancer, and melanoma.

Dr Gentile highlighted a number of demographic differences across the three data sources:

Dr Gentile summed up the implications of this research, explaining that while oncology clinical decisions rely on the outcomes from RCTs, only about 3% of oncology patients participate in such trials. This research shows that RWD includes a greater representation of minoritized populations and could be considered alongside RCT data to understand the full impact of oncology treatments [5].

Case study: Standardization in real-world study endpoints

“Fit-for-purpose data means that we have the appropriate data to address the scientific question that we are trying to assess for a given study,” Dr Richard Scott Swain, Cardinal Health.

Next, Dr Swain summarized the findings from an FDA-funded project evaluating the accuracy of RWD for lymphoma tumor response compared to blinded independent central review (BICR), the gold standard in RCTs. He began by discussing the need to standardize endpoints in RWE research to ensure it is fit-for-purpose; for example, while claims and EHR data are useful for drug utilization studies, they often fall short in capturing patient safety and efficacy outcomes, especially in oncology. However, though chart review studies – often considered the ‘gold standard’ in RWD – tend to generate deeper data than can be extracted from claims or EHR studies, accurate classification of some patient outcomes can be a challenge. Dr Swain described novel approaches to standardized RWD endpoints that emulate procedures used in clinical trials. He went on to share research on developing a novel methodology, called real-world Lugano (rwLugano), derived from the Lugano 2014 classification criteria, to assess response to diffuse large B-cell lymphoma (DLBCL) treatment in real-world studies.

Dr Swain explained the objective of the rwLugano study: to determine if rwLugano is a more accurate method for lymphoma treatment outcome classification in real-world studies compared to physician-charted response. In lymphoma, the ‘gold standard’ outcome is BICR measuring Lugano 2014 criteria. In the study, data extracted from medical charts were used to calculate rwLugano. Then, physician-charted and rwLugano-derived response were compared to BICR. They observed that the physician’s initial treatment assessment tended to underestimate response, compared to BICR; whereas rwLugano-derived endpoints were not statistically different than BICR-assessed response. Thus, the study found that rwLugano could be a better endpoint than physician-charted response for real-world lymphoma research [6].

Dr Swain explained that the purpose of the research is to establish whether methodologies such as rwLugano advance the state-of-the-art in outcome assessment. If these findings are replicated, such approaches could help address several current RWE limitations including:

  • Endpoints: Exposure-based real-world endpoints have limited association with clinical endpoints and methods to evaluate clinical endpoint, such as treatment response, lack standardization across RWD studies
  • Time: BICR is resource intensive and typically not feasible
  • Bias: Physician-charted response may be subjective and is assessed differently in clinical practice than in clinical trials
  • Standardization: Standardized approach reduces variability and misclassification and increase generalizability and reproducibility between studies

Dr Swain noted that another important takeaway from the study was the successful completion of a BICR in real-world settings. Fit-for-purpose outcomes means that in some circumstances, rigorous methods, such as BICR, may be necessary for RWD to meet standards for scientific assessment. For example, incorporating BICR assessment in prospective external control arms could increase comparability with clinical trial data. He also noted that Cardinal Health is investigating the validity of a similar novel methodology in solid tumors using RECIST 1.1 criteria (i.e., rwRECIST.)

Dr Swain concluded by reiterating the need for deep clinical data in real-world oncology research to provide meaningful, fit-for-purpose insights that provide a complete picture of real-world patient populations. Standardizing endpoints is crucial to narrow the evidence gap between RWE and clinical trials.

“The use cases for RWD research have evolved over time, from rare safety events to comparative efficacy. Our goal is to reduce the evidence gap between real-world research and clinical trials. Novel methods, such as rwLugano and rwRECIST, represent a step in that direction – increasing the comparability between studies and across data sources.”

Q&A – How is fidelity maintained within the PRN to ensure data collection is consistent and reliable?

Dr Gentile addressed this first question from the audience by clarifying the importance of consistent data collection within the PRN. All patient demographic, clinical and treatment data captured within electronic case report forms (eCRFs) by practices in the PRN is standardized. Before the eCRF is circulated to the research networks, they undergo a test cycle with physician abstractors to ensure all components of the form are clear and efficient. All data abstractors receive training before each chart review begins, allowing us to address the unique subtleties of each study. Physicians also provide iterative feedback of eCRFs, with any differences across sites monitored and clarified by the research operations team.

Q&A – How does the PRN support studies in specific patient populations?

Both presenters offered their insights into this question. Dr Gentile explained that very detailed inclusion and exclusion criteria are used to pinpoint specific patient populations. Dr Swain expanded by providing context within the RWD research process. For each study specified by a client, they identify the relevant patient population and reach out to their networks to find patients meeting the required criteria. Together with the client, they decide on the necessary data elements from patient charts to create a unique dataset for that specific research project. The research team at Cardinal Health has overseen hundreds of RWD studies, encompassing over 41,000 patient charts. Consequently, the scientific teams have fine-tuned the clinical data elements available for extraction as well as the best approaches to obtain those elements.

Q&A – How is missing data accounted for within the PRN?

Dr Swain explained a unique feature of the data abstraction process that has been fine-tuned for the PRN means the data collected is robust and is highly complete. As physicians collect data from their own patients, or those treated in their practice, they have full access to medical records, not just structured fields. Our internal team estimates that about 80% of the data collected is not in a structured format. He noted that many data points that should exist in structured fields, such as race and ethnicity, are often found in the unstructured notes, and most safety and effectiveness endpoints are found in unstructured notes, pathology reports, radiology scans, hospital records, or other medical records that would be difficult to identify except for those with intimate knowledge of that practice’s medical record system. Data quality is assessed by having physicians re-abstract random data points for verification. In addition, the long-standing relationship with these physicians within OPEN and the PRN fosters mutual trust that enables investigation for any outliers that may occur.

Learn more about Cardinal Health Real-World Evidence and Insights

About the speakers

Richard Scott Swain, Director, Regulatory Science and Real-World Evidence, Cardinal Health, USA

In his role as Director of Regulatory Science and Real-World Evidence with Cardinal Health, Richard Swain’s primary duties include developing strategies to incorporate alternative study design methods, such as inclusion of RWE, to increase efficiency of clients’ regulatory submissions. Previously, as Lead Epidemiologist in the FDA/CDER Division of Epidemiology, he led the epidemiology team overseeing RWE safety and efficacy issues for oncology, hematology, and medical imaging products. He has over a decade of experience with epidemiological study design, protocol, and statistical analysis plan development, performing systematic literature reviews, and conducting research using large, multi-level electronic healthcare datasets in regulatory, consulting, and academic settings. His publication history includes abstracts and presentations for scientific conferences and manuscripts in peer-reviewed journals. Swain earned a PhD in Epidemiology, and a Certificate in Pharmacoepidemiology, from Johns Hopkins University and an MPH in Epidemiology, with a concentration infectious disease, at the George Washington University. His additional areas of expertise include RWE in oncology, Type 1 diabetes, self-harm and suicide, and clinical trials.

Danielle Gentile, Principal Scientist, Real-World Evidence and Insights, Cardinal Health, USA

Danielle Gentile, PhD is a Principal Scientist on the Real-World Evidence and Insights team at Cardinal Health. She investigates the health outcomes of patients treated in real-world settings outside of clinical trials. Before joining Cardinal Health, she served as a Health Services Researcher in the Department of Supportive Oncology at Levine Cancer Institute, Atrium Health in Charlotte, NC where she resides. She earned her PhD in Health Promotion, Education, and Behavior from the Arnold School of Public Health at the University of South Carolina.

 

 

Sponsorship for this Deep Dive was provided by Cardinal Health

© 2024 Cardinal Health. All Rights Reserved. CARDINAL HEALTH and the Cardinal Health LOGO are trademarks of Cardinal Health and may be registered in the US and/or in other countries. All other trademarks are the property of their respective owners. Patent cardinalhealth.com/patents.

 

Source

  1. Representation of age, race and ethnicity in real-world evidence (RWE), randomized clinical trial (RCT), and Surveillance, Epidemiology, and End Results (SEER) cancer patient registry data in advanced renal cell carcinoma (aRCC)
  2. Representation of Sex at Birth in Real-World Evidence, Randomized Clinical Trial, and Surveillance, Epidemiology, and End Results (SEER) Cancer Patient Registry Data By Cancer Type
  3. Age Comparisons across Real-World Evidence Research, Randomized Clinical Trials, and Surveillance, Epidemiology, and End Results (SEER) in Advanced Cancer
  4. Comparing Representativeness of Race and Ethnicity in Real-World Evidence, Randomized Clinical Trial, and Surveillance, Epidemiology, and End Results (SEER) Cancer Patient Data
  5. The Role of Clinical Trial Participation in Cancer Research: Barriers, Evidence, and Strategies
  6. A Novel Methodology for Assessing Response to Lymphoma Treatment in Real-World Studies – the Real-World Lugano (rwLUGANO) Study