Deep learning-based prognosis models accurately predict the time to delivery among preeclamptic pregnancies using electronic health record

Data Source

We obtained the discovery cohort from Michigan Medicine (UM), the academic health care system of the University of Michigan, Ann Arbor. Data usage was approved by the Institutional Review Board (IRB) of the University of Michigan Medical School (HUM#00168171). The validation cohort was obtained from the Integrated Data Repository database at the University of Florida (UF). Data usage was approved by IRB of the University of Florida (#IRB201601899). We extracted all obstetric records with at least one PE diagnosis based on ICD-10 diagnosis codes (Supplementary Table 1). We excluded patients with the following conditions: Hemolysis, Elevated Liver Enzymes, and Low Platelet (HELLP) syndrome and eclampsia, for which iatrogenic delivery is ubiquitously induced within 48 hours of diagnosis; chronic hypertension with superimposed PE, whose onset may occur before week 20 and whose diagnostic criteria are less clear; and postpartum PE, which is only developed after delivery. We also removed patients transferred from other institutions by deleting patients with no visit record within 180 days prior to the first diagnosis of PE to confirm the accuracy of the initial diagnosis time of PE.

EHR Feature Engineering

The EHR provided baseline individual features, vital signs, and lab values obtained after PE diagnosis. Baseline features included age, race, ethnicity, smoking and drug use status, medical history, pregnancy characteristics, and comorbidities at the earliest PE diagnosis. Pregnancy characteristics included parity, number of fetuses, gestational age, and PE severity at initial diagnosis. Comorbidities were grouped into 29 categories using the Elixhauser Comorbidity Index¹⁷. The observational window for lab results and vital signs was 5 days before the day of the initial PE diagnosis. Only the first results of repeated tests were used to avoid intervention/treatment effect, resulting in 13 lab test features (10 hematological, 2 liver function, 1 urine). Vital signs were collected within the 5-day window, and summary statistics of systolic blood pressure (SBP), diastolic blood pressure (DBP), and respiratory rate (RR) measures were included (max, min, mean, standard deviation), as done in previous work¹⁸. As a result, 62 features were kept for initial analysis (Supplementary Table 2). Details of feature selection and cleaning are described in Supplementary Methods.

Fully-connected Cox-nnet neural network models

We developed 4 models to predict the time to delivery of PE patients: PE baseline, PE full, EOPE baseline and EOPE full model. The baseline models include demographics, medical history, comorbidities, the severity of PE, and the gestational age of initial PE diagnosis. The full model incorporated all features from the baseline model, with additional lab results and vital signs collected in the observation window. EOPE models were built and tested on patients with PE onset time before 34 weeks of gestation. We constructed all models using the Cox-nnet v2 algorithm (Supplementary Figure 1)¹⁴. In this study, we adopted the model to predict the time between PE diagnosis to delivery. To ensure the stability of the models, the discovery dataset was divided into a training set (80%) and a hold-out testing set (20%).

Reduced feature representation from the Cox-nnet models

To derive a subset of clinically significant and easily interpretable features, we reduced Cox-nnet features based on both their importance scores and significance levels. To do so, we first selected the top 15 most important features based on their average permutation importance scores generated by Cox-nnet models¹⁹. Then we fit each of the K features individually by a univariate Cox-PH model and kept those features with statistical significance (log-rank p-value <0·05). We rebuilt the clinically informative Cox-nnet models with the reduced set of features, exactly the same way as the models using all initial input features.

Interactive Web Application for Easy Model Validation

To disseminate the models for public use, we containerized the pre-trained Cox-nnet model into a Docker-based web application using R shiny²⁰. This allows the users to access the models easily through a local web interface and get prediction results quickly. This app contains two main panels: the individual prediction panel and the group prediction panel. Using pre-trained models, the individual prediction panel calculates the prognosis index (PI) score of a single new patient, marking its positions and percentiles in a distribution plot of PIs within the UM discovery cohort. The group panel takes in a group of new patients and returns predicted PIs and percentiles of their PIs in a table. The shiny app is available at http://garmiregroup.org/PE-prognosis-predictor/app

Role of the funding source

The sponsor of the study had no role in the study design, data collection, data analysis, data interpretation, writing of the manuscript, or decision to submit the manuscript for publication.

Cohort characteristics

The discovery cohort contains EHR records from 1,533 unique preeclamptic pregnancies, including 374 pregnancies complicated by EOPE, from Michigan Medicine between the years 2015 to 2022. Patients with HELLP syndrome, chronic hypertension with superimposed PE, and postpartum PE were removed from the cohort. Additionally, transferred patients from other clinics outside of Michigan Medicine were excluded from the cohort. Following the same inclusion/exclusion criteria, 2,249 unique preeclamptic pregnancies (547 with EOPE) from the University of Florida (UF) between 2015 and 2022 were identified as the validation cohort (Figure 1). Summaries of the patient characteristics of these cohorts are shown in Table 1 and 2.

• Download figure
• Open in new tab

The baseline prediction model of time to delivery interval among PE patients

We built the baseline model using 47 variables including patient demographics, medical history, comorbidities, PE diagnosis time, and severity (Supplementary Table 2). We randomly split the discovery dataset into a training set (80%) and a hold-out testing set (20%). We then built the survival prediction model using the Cox-nnet (version 2) algorithm¹⁴ (Supplementary Figure 1). Cox-nnet (version 2) is a multilayer perceptron prognosis prediction model based on Cox Proportional Hazards regression, suitable for EHR data prediction (Methods). The resulting model is predictive with C-indices of 0·73, 0·72, and 0·71 in the UM training, UM hold-out testing, and UF validation cohorts, respectively (Figure 2A).

• Download figure
• Open in new tab

To enhance the clinical utilities of the Cox-nnet model, we reduced the number of predictive features. We selected the top 15 most important features based on their average permutation importance scores in the Cox-nnet model¹⁹, followed by univariate Cox-PH fitting for each to keep only the features with statistical significance (log-rank p-value <0·05)¹⁹. This procedure resulted in 7 significant features, which we used to rebuild the “clinically informative Cox-nnet baseline model”. It reaches high predictability of the time to delivery, with median C-index scores of 0·73, 0·74, and 0·73 on UM training, hold-out testing, and UF validation dataset respectively (Figure 2A). We stratified patients into 3 risk groups by the quartiles of predicted PI scores from the reduced model: high-risk (upper quartile), intermediate-risk (interquartile), and low-risk (lower quartile) groups. The survival curves of the time to delivery interval on these three risk groups display significant differences (log-rank p-value < 0·0001) on both the hold-out testing set (Figure 2B) and validation set (Figure 2C), confirming the strong discriminatory power of the PI score.

The seven features in the clinically informative baseline model included those that shorten the time to delivery and extend the time to delivery (Figure 2C; Table 3). In descending order of importance scores, the features that shorten the time to delivery are: gestational age at diagnosis, sPE, uncomplicated pregestational diabetes mellitus, and parity. Conversely, features extending the time to delivery are: PE in a prior pregnancy, increasing maternal age, and comorbid valvular disease. To demonstrate the associations of these important features with time to delivery, we dichotomized patient survival in the hold-out testing set by the median value of each feature (Supplementary Figure 2). All features, except maternal age, show significant differences (log-rank p-value < 0·05) between the dichotomized survival groups. We further examined the relationship of the top 3 features (gestational age at diagnosis, sPE, and history of PE in prior pregnancy) with the gestational age at delivery and time to delivery (day) using the UM discovery set in (Figure 2E-2J). Later gestational age at diagnosis leads to a later gestational age of delivery (Figure 2E), but a shorter time to delivery (Figure 2H). Higher percentage of patients with earlier gestational age of delivery (Figure 2F) and shorter time to delivery (Figure 2I) are diagnosed with sPE. In the deliveries from smaller (<32 weeks) gestational ages, the percentages of patients with PE in prior pregnancies are significantly higher (Figure 2G). However, the percentages of prior PE fluctuate with respect to time to delivery (Figure 2M).

The full model of time to delivery among PE patients

We next investigated the contribution to time of delivery from all 62 variables, including the 47 baseline variables above and an additional 15 laboratory testing results and vital signs obtained in the 5-day observation window before the time of diagnosis (Supplementary Table 2). We performed model construction, validation, and feature reduction for clinical use in the same way as the baseline model. As a result, 12 top features were kept in the clinically informative full model (Table 4). This model shows higher predictive accuracy of time to delivery compared to the seven-feature baseline model, with median C-index scores of 0·78, 0·79, and 0·74 in the training, testing, and validation datasets respectively. The difference between survival curves of the high-, intermediate- and low-risk groups stratified by predicted PI scores from the reduced model has log-rank p-values close to 0 in the hold-out test set (Figure 3B) and validation set (Figure 3C), even more significant than those from the baseline model (Figure 2B and 2C).

• Download figure
• Open in new tab

Further examination of the 12 important features in the full model (Figure 3D, Table 4) shows good consistency with the 7-feature baseline model (Figure 2D, Table 3). Five out of seven features in the baseline model also exist in the full model with similar importance scores: gestational age at diagnosis, sPE, parity, maternal age, and PE in prior pregnancies. Gestational age at PE diagnosis and sPE continued to be the two most important features in the full model. New features associated with shorter intervals from diagnosis to delivery were also identified. They were, in descending order of feature importance: aspartate aminotransferase (AST) value, the standard deviation of diastolic blood pressure (DBP), the standard deviation of respiratory rate (RR), creatinine value, mean DBP and white blood cell count (Figure 3D). Conversely, platelet count was identified as a new feature with a negative importance score, which means it’s associated with a longer time to delivery. All dichotomized survival plots using median stratification on each of the 12 important features have log-rank p-values smaller than 0·05, confirming their associations with time to delivery in the discovery set (Supplementary Figure 3). We examined the 3 top lab/vital sign features: AST, the standard deviation of DBP, and the standard deviation of RR, on their association with the duration of time between diagnosis and delivery. These values show negative trends with time to delivery, particularly for AST value and the standard deviation of DBP (Figure 3E-G). These 3 features are roughly uniform across delivery gestational ages, except AST which shows slightly higher values in deliveries less than 32 weeks of gestational age (Supplementary Figure 4).

Time to delivery prediction of EOPE patients

Accurate prediction of EOPE patients’ time to delivery is crucial, given that delivery of a premature infant has more significant neonatal consequences. Using similar modeling techniques, we trained two additional EOPE-specific Cox-nnet (version 2) models (baseline vs. full model), using the same features described earlier (Supplementary Table 2), on a subset of 374 EOPE patients from the UM discovery cohort.

The C-indices for the clinically informative EOPE baseline model are 0·67, 0·68, and 0·63 on the UM training, testing, and UF validation sets, respectively (Figure 4A). In the UM hold-out testing set and UF validation set, the high-median-, and low-EOPE patient delivery risk groups using the PI scores from the reduced model show significant differences, with log-rank p-value < 0·001 (Figure 4B and 4C). This baseline model consists of the six most important features: gestational age at diagnosis, sPE, PE in a past pregnancy, parity, pulmonary circulatory disorders, and coagulopathies (Figure 4D; Table 5). All survival plots, dichotomized using the median stratification on each of the 6 features, have log-rank p-values smaller than 0·05 in the discovery dataset (Supplementary Figure 5).

• Download figure
• Open in new tab

The clinically informative EOPE full model reached much higher accuracy compared to the EOPE baseline model, with median C-indices of 0·74, 0·76, and 0·67 on the training, testing, and validation sets (Figure 4E). The 3 risk-stratified groups within the EOPE patients cohort also showed significant (log-rank p-value<0·001) differences in the hold-out testing set and validation set (Figure 4F, 4G). This model contains 13 important features selected from the original 62 features (Figure 4H; Table 6).

Gestational age at diagnosis continued to be the most important feature. Several other features (eg. PE with severe symptoms, PE in a past pregnancy, parity, and coagulopathy) were of significant importance as well, similar to the EOPE baseline model. Many additional features in the vital signs and lab test categories were also significant, including creatinine value, mean DBP and mean SBP, standard deviation of RR, AST, and platelet counts. Among these 13 features, parity, PE in a prior pregnancy, and higher platelet counts were protective against early delivery (Figure 4H).

We created dichotomized survival curves based on creatinine value and platelet counts, two new features relative to the EOPE baseline model. Both of them show strong distinctions between the risk groups (Figure 4I, 4L), similar to all other selected features (Supplementary Figure 6). These two features also revealed systematic trends in associations with the gestational age at delivery and time from diagnosis to delivery. Patients with high creatinine levels were more likely to be delivered within 3 days or less of diagnosis and more likely to deliver preterm (Figure 4J, 4K). Lower platelet counts were also associated with shorter time to delivery (Figure 4N), even though the platelet levels were not strongly associated with gestational age at delivery among all EOPE patients (Figure 4M).

PE time to delivery predictor graphic user interface (GUI)

To disseminate our model publicly, we packaged the pre-trained clinically informative models above into an interactive, user-friendly web application using R shiny²³. We named this app “PE time to delivery predictor”. The app contains two main panels: the single-patient prediction panel and the group prediction panel (Supplementary Figure 7). The single-patient prediction panel calculates the prognosis index(PI) of a single patient if provided the required clinical variables. The PI score describes the patient’s risk of delivery at the time of the diagnosis of PE, relative to the population. The panel also provides the percentile of the PI score among the training data, and displays the results in a histogram figure and a table. The group prediction panel calculates the PI and PI percentile of multiple patients simultaneously and also displays them in a table, below the histogram built on the training data. The app is available at http://garmiregroup.org/PE-prognosis-predictor/app