Methodological Challenges using Routine Clinical Care Data for Real-World Evidence: a Rapid Review utilizing a systematic literature search and focus group discussion

Data sources and search strategies

In October 2023, we conducted a systematic search of MEDLINE via PubMed. The search focused on English-language publications in the following top-ranked journals based on the Journal Citation Reports (JCR) category “Medicine, General & Internal” as of 2022 [16]: (i) The Lancet, (ii) New England Journal of Medicine, (iii) Journal of the American Medical Association (JAMA) and (iv) British Medical Journal (BMJ). In addition, we included the following oncology journals, based on our experience, as RWE is particularly prevalent in this therapeutic field [17]: (v) JAMA Oncology, (vi) The Lancet Oncology, and (vii) Journal of Clinical Oncology. In order to ensure recency and thereby relevance of the articles, the search included all original articles published between January 2018 and October 2023. The following terms were queried in the title or abstract: “real-world evidence”, “RWE”, “Real-world data”, “real-world”, “routine data”, “routine care data”, “Emulation” and “Electronic health data”. A study was defined as being eligible if routine clinical care data collected by the healthcare system, such as longitudinal claims data or EHR, was analyzed. Other types of publications, such as comments, letters, perspectives or reviews were excluded. Studies were also excluded, in which only registry data or manually collected data were analyzed. In addition, we assessed if any aspect from the three pre-defined scenario categories were investigated in the studies: (1) Burden of Disease, (2) Safety and Risk Group Analysis, and (3) Treatment Comparison. If this was not the case, the study was excluded. The detailed search strategy is described in the Supplementary Material (Supplementary S1). All articles were indexed and organized using Zotero.

Three reviewers (MP, KG, KU) independently screened each publication through all stages of the review process. First, titles and abstracts of the search results were screened to ensure relevance and adherence to the inclusion and exclusion criteria. In a second step, the full texts of the included abstracts were assessed to obtain a final decision on inclusion in the review. In case of any discrepancy between two reviewers regarding the eligibility of specific studies, an additional independent reviewer (MB) was consulted to resolve the issue. The review was pre-specified and registered in PROSPERO (CRD42023477616).

Data extraction and categorization

Methodological challenges and limitations mentioned in the discussion sections of the included publications were extracted as the main outcome. In addition, study characteristics such as design, the underlying data sources, the country from which data was obtained, the methods employed, and the purpose of the published studies were retrieved. Publications were categorized into the three predefined main areas of application (Burden of Disease, Safety and Risk Group Analysis, Treatment Comparison), based on their research question. The extracted limitations were subsequently assigned to the main bias categories in non-randomized studies as defined in the Cochrane Handbook: confounding, selection bias, information bias and reporting bias [18]. We identified five specific categories to highlight potential biases that are specific to the use of routine data. First, as clinical routine data is primarily collected for reimbursement purposes based on clinic-specific coding practices, the category Coding Challenges is introduced. This category includes for example discrepancies in coding practices between clinics or specificalities of the ICD10 (the International Classification of Diseases and Related Health Problems) coding system [19]. This challenge may lead to information bias, specifically misclassification or detection bias, regardless of the already well-known detection bias, which typically result from varying quality in detection methods. Second, as one of the main drivers of documentation quality is again reimbursement as well as patient care rather than research, the category Operationalization or Availability of Variables is established to address to which extent studies were restricted by the availability of the data for answering the research question. This type of missing data may lead to unmeasured confounding and potential selection bias. Third, routine data may suffer from other types of missing data, i.e., missing records in certain variables, as typically known from observational studies, resulting in the category Missing Data. Fourth, routine data may also suffer from different lengths of follow-up largely varying between patients, for which we introduce a further category called Follow-up Challenges. Finally, the category Validation & Data Quality is added, which leads to potential bias as large amounts of routine data typically cannot be carefully validated. This also includes the variability of data quality over time arising e.g., due to changes in coding systems, such as transitions between versions of the ICD coding standard, or the ongoing digitalization of hospitals.

We did not assess the quality or risk of bias of individual studies since we were not extracting data or outcomes of the studies for subsequent analysis and we were only interested in their stated limitations. Still, we expected that there were challenges not reported in the study publications, e.g., because of limited relevance or lack of awareness. Therefore, a subsequent focus group meeting was initiated for complementing and ranking the identified list of challenges.

Focus Group

At a four-hour workshop, the results of the systematic literature search were presented in the form of a slide presentation for each scenario to a selected group of co-authors of this manuscript plus one independent expert from the area of health technology assessments. After each scenario was presented, focus group members were asked to add potential additional challenges in a moderated joint discussion. In addition, through a General category, we left room for limitations and challenges that can be found in RWD studies not specific to any of the three scenarios. All identified challenges were listed for a subsequent voting on their potential impact within each scenario. Every focus group member had two votes for each of the three main scenarios, plus one additional vote for the General category. All challenges with at least one vote were summarized and considered as key challenges deemed to have a high potential to influence the results requiring thorough consideration in analyses.

Study selection and characteristics

In total, 227 records were identified from MEDLINE of which 15 duplicate publications and 1 audio interview were removed, leading to 211 records eligible for title and abstract screening. From this first screening, we excluded 142 publications because they were not original research articles, were only methods articles, or did not use routinely collected data from clinical care. In a second step, 69 records were evaluated in a full-text screening for verifying the data source as coming from electronic health records or administrative data, and to be categorizable into the three main scenarios. Finally, 39 studies were included in the review. An overview of the review process and study selection is depicted in Figure 2.

• Download figure
• Open in new tab

Figure 2 PRISMA flow diagram [20]

A brief summary of the study characteristics is shown in Table 1. Most of the studies were published in BMJ (36%), with 55% being published in the past two years (2022 and 2023). This is supporting our decision to limit the search to the past five years. The majority of extracted publications (46%) was based on data from the US. While some publications used only EHR data (59%), other studies linked this information to additional sources, such as biobanks or registries (28%). The use of data sources varied between studies, from single-country to multinational studies. The majority of the publications used nation-wide (46%) or multi-center data (38%), while the remaining studies used multi-national (10%), single-center (3%) and territory-wide (3%) data.

Figure 3 provides a summary of the limitations mentioned in the discussion section of the extracted studies stratified into the data source types and according to their assigned scenario category. In total, there are 11 publications in the category Burden of Disease, 17 publications in Safety and Risk Group Analysis and 11 publications in Treatment Comparison. As expected, the main bias types are typically mentioned in the publications. However, it is important to recognize the additional biases that frequently appear. A detailed overview for each publication is provided in the Supplementary Material S2 – Table 3, including a broad overview of the study design including the main outcomes and their respective methods, objectives, and the scope of the analysis in terms of data use.

• Download figure
• Open in new tab

Figure 3 Summary of biases extracted from the studies

Number of studies facing the main biases and additional identified bias categories grouped by data source types and the main scenarios: Burden of Disease, Safety and Risk Group Analysis, and Treatment Comparison. Abbreviations: EHR…Electronic Health Records, Admin. Data… Administrative Data

In the following, we summarize the findings on methodological challenges and limitations from both the review and the focus group workshop for each main scenario, separately. In Figure 4, all challenges that were added and ranked according to their relevance in the focus group discussion are summarized.

• Download figure
• Open in new tab

Figure 4 Overview of Workshop Results

Findings of the focus group discussion divided into the three main scenarios: Burden of Disease, Safety and Risk Group Analysis, and Treatment Comparison, as well as the additional category General Limitations. The content stated within the blue colored areas are the voting result, with the plus signs (+) indicating the importance weighting. The blue font highlights challenges and limitations already occurring in the systematic search. The black font and particularly the content outside the blue area are limitations and challenges added by the focus group.

Burden of disease

Five publications reported limitations potentially leading to confounding. Of these, two studies had limited availability of variables included in the analysis resulting in the possibility of unmeasured confounding [21,22]. However, Kamran et al. [22] argued that with the use of a limited number of variables in the model a reliable identification and validation of all variables even across different institutions was feasible. Limitations related to the risk of selection bias were present in eight studies. On the one hand, in Canavan et al. [21] a volunteer bias could not be ruled out because the data used for the analyses included individuals who opted to be included in the study database. On the other hand, in Witberg et al. [23] the absence of simultaneously enrolled comparison groups was mentioned which could lead to a potential selection bias. Moreover, some studies highlighted the fact that the results might not be transferable to other clinical practices or populations [21,24–26]. The primary type of information bias (n=7) observed was misclassification bias, affecting either the exposure or outcome [21,24,27,28]. Beck et al. [24] highlighted that clinical diagnoses were probably not fully captured due to challenges in diagnostic and physician coding. On the other hand, Manz et al. [28] noted a limitation indicating that they were limited to use only patients with a coded classifier variable for comparing the machine learning model with the commonly used prognostic reference. This limitation resulted from differences in the characteristics of the patients with and without the coded variable. Kamran et al. [22] additionally emphasized that even though a common EHR provider across facilities was implemented, it remains crucial to have in-depth knowledge of the local deployment. Two studies reported limited follow-up time [26,29]. However, Biccler et al. [29] noted that despite the longer follow-up period compared to clinical trials, it was still insufficient to evaluate very long-term effects. The lack of health record validation was reported in two studies [29,30]. While all studies acknowledged limitations in their discussions, only three studies [22,25,27] took an additional step to assess these limitations in the context of their specific study.

During the focus group discussion, further limitations were added to those identified in the literature review and are presented in the first part of Figure 4. Some relevant and reasonable limitations added by the experts were the inconsistency of the data quality over time. This challenge is interconnected to the adjustment or calibration of algorithms to dynamic data changes, especially when using a model that continuously uses recent data from routine clinical care. Additionally, the heterogeneity and specialization of hospitals substantially impact the burden of disease. Including only hospitals with specializations in a certain disease treatment would overestimate the burden. Another added limitation was missing information due to death in situations with death as competing risk, where it would be impossible to diagnose the disease even though the patient may have had it [31].

The rating of the limitations of Burden of Disease is indicated by the blue area in Figure 4. The transferability of study results to a larger population, which goes hand in hand with the possibilities of selection bias, missing data and misclassification of data, was weighted by the experts to be most important. The importance of data quality in EHR data was also recognized, specifically the variability of data quality over time due to e.g., changes in coding systems, such as transitions between versions of the ICD coding standard, or individual coding behavior. This could potentially introduce bias to the results concerning the burden of a disease, particularly when comparing diagnoses from the present to the past or the burden over an extended period of time.

Safety and Risk Group Analysis

In this category, the information bias was predominantly manifested as the risk for misclassification bias [32–40] with only one study addressing immortal time bias [38]. Specifically, certain limitations were connected with coding challenges, since the classification was based on diagnosis codes from the EHR systems [33,35]. In addition, some form of potential information bias occurred because the drug use in an EHR system had been defined as the prescription of the medication and not as the actual drug intake [36,41]. Aspects like the quantification and analysis of information concerning the physician’s decision-making process for treatment selection, treatment switching or the effect of pre-treatment were often missing or inadequately explored [38,40,42] which could lead to confounded results. Details regarding adherence or dosage, as well as unstructured free-text information, were not accessible [40,42,43]. Additionally, significant risk factors, such as smoking, were not available, creating a potentially high risk of confounding in the analysis. Selection bias limitations were evident in ten studies including volunteer bias due to the involvement of individuals who volunteered to join the database used for analysis [35,44]. Moreover, several studies highlighted site heterogeneity, noting that certain hospitals with robust programs may have had a higher burden of disease [33–35]. The limitation of missing data was mentioned in five studies [33,36,38,39,45], for instance, in Li et al. [36] who stated that EHR data sources lacked comprehensive coverage of medical events recorded in other healthcare facilities. While only three studies [38,43,44] explicitly stated their limitations without providing further elaboration, the remaining studies either addressed how they mitigated certain limitations through selected methodologies or sensitivity analyses [34,36,37,39–42,45,46]. Alternatively, they presented arguments explaining why certain limitations were unlikely or invalid in the context of their specific study [32,33,35,39,45,47].

A few specific challenges were added by the experts in the focus group discussion as displayed in the second section of Figure 4. First, it is considered important to correctly distinguish between the case and patient definition. Given that patients may have multiple cases associated with them in routine clinical data, it is essential for the research question to specify whether the data refers to the patient level or the individual case level. The lack of clarity could potentially violate the i.i.d-assumptions (independent and identically distributed assumption), which is the basis for many commonly used statistical methods.

Second, another challenge mentioned is the problem of reverse p-hacking for safety endpoints, where the analysis is manipulated to intentionally favor non-significant results [48]. Third, the focus group emphasized the importance of the limitation related to the lack of follow-up data, especially in health care systems where no linkage with routine outcome information is available. A longer follow-up period is crucial for conducting thorough safety analyses to observe adverse events. This aspect is closely linked to the issue of missing information due to death, as death prevents the observation of subsequent adverse events.

Moreover, the experts added several limitations related to medication use. These included insufficient information on the reasons for change in medication and its effect, as well as information on dosage. Additionally, concerns were raised about the off-label use of medication. Unlike in controlled clinical trials, the treatment paths of patients are quite heterogeneous due to the decisions of the clinician or the different specializations of the hospitals.

Treatment Comparison

Confounding remained a particular area of concern, specifically in achieving balance between the groups. Kim et al. [49] reported persistent imbalance in variables even after applying matching techniques. However, they addressed this issue by adjusting for these variables in further analyses. Likewise, Xie et al. [50] stated that individuals treated with the study medication had a higher baseline health burden, potentially leading to underestimated findings due to residual confounding. Only two studies explicitly report missing risk factors that could subsequently not be used for balancing [51,52]. Wang et al. [8] highlighted that claims data had not recorded medication use in hospital, therefore, they needed to use alternative index date and follow-up definition. Similar to the Safety and Risk Group Analysis, most potential information biases were a type of misclassification [50,53–56]. Rentsch et al. [52] reported that the identification of outcome events was not provided by a validated algorithm, which could also potentially lead to information bias. Same holds true for Wong et al. [57] as they did not distinguish the reason for deaths. Kim et al. [49] emphasized the general issues of immortal time and time lag biases in the discussion section, but elaborated on the methods used to address and avoid these biases. One study noted the concern of potential upcoding, i.e. the intentional use of ICD-10 and OPS (Operation and Procedure Classification System) codes with the greatest reimbursement rather than those with the greatest clinical relevance, resulting in overstating the patient’s severity of disease [58]. Xie et al. [55] highlighted that the effectiveness of the investigated drug could be overestimated in individuals with poor health characteristics who opt not to receive treatment, while those with better health characteristics who decide not to be treated could lead to underestimation. The studies predominantly stated their open limitations including those that persisted despite implementing causal inference methods or conducting sensitivity analyses aimed at addressing them [50,52,53,55]. Beyond that, Kim et al. [49] detailed how they mitigated their limitations.

The focus group discussion resulted in the following additions. First, data availability with regard to the transferability within different health care sectors, e.g., ambulatory and stationary sector, was added as challenge, especially for Germany [59]. Second, the timing of an intervention is also critical to assess its effectiveness accurately, understand its impact on patient outcomes, and make informed clinical decisions. In EHR or administrative databases, the timing may not be explicitly recorded or may be subject to errors or omissions. Third, the difficulty in defining control groups for comparison was highlighted as well as the verifiability of the inclusion or exclusion criteria [9,60]. Fourth, similar to the Safety and Risk Group Analysis scenario, follow-up duration is critical. However, in the context of treatment comparison, the criticality arises also from the possibility that drugs can have a longer delay in efficacy.

Compared to the previous scenarios the ratings of the Treatment Comparison category were more widely distributed, with votes spread across numerous limitations. This distribution highlighted the wide range of challenges to consider. Residual confounding and immortal time bias received the majority of votes. Not only the data availability of different data sectors, but also the availability of variables was considered important which is connected to residual confounding, e.g. absence of known confounders.

General limitations and challenges

The focus group viewed the linkage of records, the validation of data and the selection of measurements as critical to conduct routine care data analysis. If there are multiple measurements, it can be challenging to specify which measurement is considered as the relevant one. In connection with record-linkage, both with other hospitals and other data sources, the cases and patient data may overlap between hospitals and other data sources. In addition, the challenges with the definition of estimands and estimators was highlighted [14,61,62].

Major findings of this work briefly summarized

This review aimed to present a comprehensive overview of the limitations and biases inherent in the analysis of routine clinical care data. Additionally, we pinpointed the challenges with the greatest potential to influence the results of the analysis emphasizing thorough consideration and discussion for the derivation of meaningful inference. We have intentionally provided a comprehensive overview of the study’s features to serve as a guide, categorically organized into different scenarios.

Strength and limitations of this work

Particular strengths of this work are, first, the inclusion of high impact journals in the review to focus on the databases with highest potential for routine data analyses and use of a broad search strategy to cover the work analyzing data routinely collected within the healthcare system. Second, we extracted and categorized the limitations and potential biases identified in the publications. In addition, our two-step procedure, which included a subsequent focus group evaluation, resulted in a more comprehensive overview of potential challenges. Finally, this review only included studies that were successfully published. Therefore, we could not identify challenges that prevented successful data analyses and publication.

Naturally, this study is also subject to further limitations. Firstly, by restricting our search strategy to high-impact publications for the sake of recency and relevance, there is a possibility that we overlooked certain publications and their associated limitations. However, the incorporation of expert experiences and knowledge helped to cover any missed challenge. Second, given that our panel consisted solely of German experts in the field of statistics and clinical epidemiology, the added limitations might be influenced by local challenges specific to this context and might miss the important aspects such as the international or the clinical perspective.

Our study assessed various limitations faced across different study scenarios of interest, resulting in a ranked list of potentially relevant challenges. While our focus was specifically on limitations stated in the discussion section of publications, we did not provide explicit methods to address these limitations. However, this has already been partially discussed in other works [10,63]. The choice of the applied methods of causal inferences, such as propensity score techniques or the method of target trial emulation for established treatments, serves to reduce potential biases and preventing fundamental errors in the study design. In the next step, we aim to explore the requirements and concerns of various stakeholders regarding the use of routine clinical data to achieve different objectives, particularly in therapeutic evaluations. This will involve understanding the diverse expectations, needs, and potential challenges faced by stakeholders in leveraging routine clinical data for meaningful and accurate therapeutic assessments.