Abstract
Multimodal learning has the potential to improve clinical prediction by integrating complementary data sources, but the incremental value of imaging beyond structured electronic health record (EHR) data remains unclear in real-world settings. We developed a multimodal survival modeling framework integrating optical coherence tomography (OCT) and EHR data to predict time to visual improvement in patients with diabetic macular edema (DME), and evaluated how different ophthalmic foundation model representations contribute to prognostic performance.
In a retrospective cohort of 973 patients (1,450 eyes) receiving anti-vascular endothelial growth factor therapy, we compared multimodal models combining 22,227 EHR variables with 196,402 OCT images, with OCT embeddings derived from three ophthalmic foundation models (RETFound, EyeCLIP, and VisionFM). The EHR-only model showed minimal prognostic discrimination (C-index 0.50 [95% CI, 0.45–0.55]). Incorporating OCT improved performance, with the magnitude of improvement depending on the representation. EHR+RETFound achieved the strongest performance (C-index 0.59 [0.54–0.65]), followed by EHR+EyeCLIP (0.57 [0.52–0.62]) and EHR+VisionFM (0.56 [0.51–0.61]). Multimodal models, particularly EHR+RETFound, demonstrated improved risk stratification with clearer separation of Kaplan–Meier curves.
Partial information decomposition revealed that prognostic information was dominated by modality-specific contributions, with OCT and EHR providing largely distinct signals and minimal shared information. The magnitude of OCT-specific contribution varied across foundation models and aligned with observed performance differences.
These findings indicate that OCT provides complementary prognostic value beyond structured clinical data, but gains are modest and depend strongly on representation choice. Our results highlight both the promise of multimodal modeling for personalized prognosis and the need for rigorous, context-specific evaluation of foundation models in real-world clinical settings.
Competing Interest Statement
C.X.C. receives equipment support from Optomed USA, Inc outside the scope of this work. M.A.S. receives contracts and grants from US Department of Veterans Affairs, Johnson & Johnson and Gilead Sciences outside the scope of this work. A.Y.L. reports personal fees from Astellas, personal fees from Genentech, personal fees from Johnson and Johnson, personal fees from Alcon, personal fees from Apellis, non-financial support from iCareWorld, non-financial support from Topcon, grants and non-financial support from Carl Zeiss Meditec, non-financial support from Optomed, non-financial support from Heidelberg, non-financial support from Microsoft, non-financial support from Amazon, and non-financial support from Meta outside the submitted work. The remaining authors declare no competing interests.
Funding Statement
S.S. contributed to conceptualization, methodology, analysis and interpretation of results, and manuscript writing. C.X.C. contributed to data curation of visual acuity data and manuscript review and editing. R.F. contributed to data extraction and manuscript editing. S.Y. contributed to OCT image extraction and data curation. D.T. contributed to data curation of visual acuity data. P.K.R., C.S.L., and A.Y.L. contributed to clinical interpretation and manuscript review. M.A.S. and Y.W. contributed to methodology and manuscript review and editing. L.Z. contributed to conceptualization, methodology, supervision, and manuscript review and editing. All authors reviewed and approved the final manuscript.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The study was conducted in accordance with the Declaration of Helsinki and was approved by the Institutional Review Board of Washington University School of Medicine in St. Louis (IRB No. 202502158). A waiver of informed consent was granted because of the retrospective design and use of de-identified data.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
