ABSTRACT
Importance Temporal dataset shift associated with changes in healthcare over time is a barrier to deploying machine learning-based clinical decision support systems. Algorithms that learn robust models by estimating invariant properties across time periods for domain generalization (DG) and unsupervised domain adaptation (UDA) might be suitable to proactively mitigate dataset shift.
Objective To characterize the impact of temporal dataset shift on clinical prediction models and benchmark DG and UDA algorithms on improving model robustness.
Design, Setting, and Participants In this cohort study, intensive care unit patients from the MIMIC-IV database were categorized by year groups (2008–2010, 2011–2013, 2014–2016 and 2017–2019). Tasks were predicting mortality, long length of stay, sepsis and invasive ventilation. Feedforward neural networks were used as prediction models. The baseline experiment trained models using empirical risk minimization (ERM) on 2008–2010 (ERM[08-10]) and evaluated them on subsequent year groups. DG experiment trained models using algorithms that estimated invariant properties using 2008–2016 and evaluated them on 2017– 2019. UDA experiment leveraged unlabelled samples from 2017–2019 for unsupervised distribution matching. DG and UDA models were compared to ERM[08-16] models trained using 2008-2016.
Main Outcome(s) and Measure(s) Main performance measures were area-under-the-receiver-operating-characteristic curve (AUROC), area-under-the-precision-recall curve and absolute calibration error. Threshold-based metrics including false-positives and false-negatives were used to assess the clinical impact of temporal dataset shift and its mitigation strategies.
Results In the baseline experiments, dataset shift was most evident for sepsis prediction (maximum AUROC drop, 0.090; 95% confidence interval (CI), 0.080-0.101). Considering a scenario of 100 consecutively admitted patients showed that ERM[08-10] applied to 2017-2019 was associated with one additional false-negative among 11 patients with sepsis, when compared to the model applied to 2008-2010. When compared with ERM[08-16], DG and UDA experiments failed to produce more robust models (range of AUROC difference, −0.003-0.050).
Conclusions and Relevance DG and UDA failed to produce more robust models compared to ERM in the setting of temporal dataset shift. Alternate approaches are required to preserve model performance over time in clinical medicine.
Question Can algorithms that estimate invariant properties across environments for domain generalization and unsupervised domain adaptation improve the robustness of machine learning-derived clinical prediction models to temporal dataset shift?
Findings In this cohort study using 4 clinical outcomes, domain generalization and unsupervised domain adaptation algorithms did not meaningfully outperform the standard model training algorithm – empirical risk minimization – in learning robust models that generalize over time in the presence of temporal dataset shift.
Meaning These findings highlight the difficulty of improving robustness to dataset shift with purely data-driven techniques that do not leverage prior knowledge of the nature of the shift and the requirement of alternate approaches to preserve model performance over time in clinical medicine.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
There was no funding support for this study.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
As this study was conducted on de-identified electronic health records, Ethics Committee/Institutional Review Board approval was waived.
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
The data used for this study is publicly available. Instructions for access can be found at https://mimic.mit.edu/iv/. The code for analyses is open source and available at https://github.com/sungresearch/mimic4ds_public.