An Interpretable Longitudinal Preeclampsia Risk Prediction Using Machine Learning

INTRODUCTION

Preeclampsia is a pregnancy-specific disorder characterized by new onset hypertension and proteinuria after 20 weeks gestation¹ that complicates 2-8% of pregnancies.^2,3 In addition to elevated systolic (>140 mmHg) or diastolic (>90 mmHg) blood pressure, patients can develop progressive end-organ damage characterized by proteinuria, elevated liver enzymes, pulmonary edema, seizures, and death. The only definitive treatment is delivery, and as a result, preeclampsia is the leading cause of iatrogenic preterm birth with associated neonatal morbidity and mortality.

Despite extensive research, the upstream causative factors for preeclampsia remain unknown. It is hypothesized that abnormal placental development may be a large contributor.⁴ Current clinical practice is focused on identifying patients at risk based on established clinical criteria, which include demographic characteristics and medical history.¹ Maternal risk can be mitigated by close surveillance using frequent home blood pressure monitoring and prophylaxis with low-dose aspirin based on the presence of at least one high-risk or two moderate-risk factors.⁵ However, more than half of pregnancies, in particular those in nulliparous individuals, end with preeclampsia in the absence of risk factors.^6,7 In addition, little is known about the trajectory of risk and the rate of change in risk across pregnancy.

As tools to predict preeclampsia are lacking, multiple predictive models have been developed. Most models utilize clinical risk factors combined with biomarkers, which may not be available in routine care.^6,8 The majority of these models are also derived from well-curated datasets using statistical methods. ^6–9 Recently, a few accurate machine learning models have been developed;⁹ however, the full potential of electronic health record (EHR) data has not been realized. Models developed to date have shown promise using small, constrained data sets but have been lacking in predictive power, accessibility, and generalizability. Machine learning models based on the entire EHR have outperformed the rule-based algorithm that relies on established clinical criteria.^10,11

The modern EHR stores a wealth of patient data and is in use across most health care institutions in the US. In large healthcare systems with multiple hospitals and outpatient offices, using uniform EHR software allows for the detailed aggregation of patient data collected at different encounters, sometimes across different hospitals or outpatient offices, longitudinally. Using the readily available rich data source of the EHR and machine learning to integrate data with complex relationships over time, we sought to develop accurate predictions of the risk for preeclampsia throughout pregnancy. We created a large patient cohort using longitudinal data from six hospitals from a large healthcare system in the Northeastern US. We developed a system of multiple machine-learning models throughout pregnancy and investigated their performance in preeclampsia prediction. Implementing this type of multi-model system to automatically operate within the EHR by ingesting data as it becomes available and generating predictions longitudinally during pregnancy would allow physicians to timely identify, triage, and modify surveillance and levels of care for the highest-risk patients.

METHODS

RESULTS

Longitudinal development of multiple predictive models

To develop predictive models of the preeclampsia risk over time, eight datasets were created at multiple time points in pregnancy from two tertiary and four community hospitals (Cohort 1); the patients from two community hospitals were reserved for external validation (Cohort 2) and those data were excluded from model training (Fig. 1A). The characteristics of both cohorts and all datasets are summarized in the appendix (pp 4-23). For all datasets that were used for model development, data were only included if recorded before the respective cut-off gestational age or before the diagnosis of preeclampsia. The data from patients without preeclampsia were censored at the respective timepoint in pregnancy or before delivery.

• Download figure
• Open in new tab

Fig. 1

Overview of the data and model development. A. After excluding patients with no data at 14 weeks, the final study patient cohort included 120,752 deliveries; of those, Cohort 1 was divided into training and testing groups used to develop the models, while Cohort 2 was reserved for external validation and did not participate in model development. B. We developed datasets at 14, 20, 24, 28, 32, 36, 39 weeks gestation, and on admission for delivery based on the timing of the routinely scheduled outpatient visits. C. After selecting relevant to the preeclampsia risk features, we developed linear regression, elastic net, random forest, xgboost, and deep neural network models to predict the risk of preeclampsia; the performance of those models was evaluated in the test datasets from Cohort 1 and external validation datasets from Cohort 2.

We developed five types of models: xgboost, deep neural networks, elastic net, random forest, and linear regression at each of the eight different time points: 14, 20, 24, 28, 32, 36, 39 gestational weeks, and on admission (Fig 1B and 1C). The earliest timepoint for which we developed each of the models was 14 weeks, as most patients had their first prenatal visit at that time, and commonly, new vital signs, detailed history, and baseline laboratory tests were obtained. We used the area under the curve (AUC), a widely used metric in machine learning that balances sensitivity and specificity, to evaluate the predictive power of the models.

We analyzed the AUC of each model using the test dataset, which included 20% of the patients in Cohort 1, whose data was not used in the process of model development (Fig. 2A). Overall, the models had AUC 0.73-0.76 at 14 gestational weeks. With the advancement of the gestational age, more data is collected from every patient visit, and the performance of each model type increased; the highest AUC was on admission, range 0.88-0.91. The detailed performance metrics are listed in the appendix (p 24). We externally validated the performance using data from two community hospitals (Fig. 2B and appendix pp 25-26). We also investigated racial bias and optimized the performance of the models (appendix pp 27-28).

• Download figure
• Open in new tab

Fig. 2.

Performance as defined by the area under the curve (AUC) of the xgboost, deep neural network, elastic net, random forest, and linear regression models to predict preeclampsia risk A. Performance on the test dataset from Cohort 1 B. Performance on the external Cohort 2.

Time course and relationships of the most predictive variables

Using Shapley additive explanations method, we evaluated the most predictive variables in the xgboost model, over the eight time points (Fig 3A). In early pregnancy, at 14 weeks, the most predictive features were chronic hypertension and long interpregnancy interval. With advancing gestational age, the highest contributing features become systolic and diastolic blood pressure. Similarly, in early pregnancy, the category that contributed the most to the model predictions was medical history; with advancing gestational age, the vital signs and laboratory results became more contributory (Fig. 3B). This approach can be used to highlight individual patient predictions and the associated factors (appendix pp 29-30).

• Download figure
• Open in new tab

Fig. 3.

Change in relative contribution for model prediction over time A. Contribution of several feature groups B. The most predictive variables over every timepoint. Labs, laboratory results; bps, systolic and diastolic blood pressure measurements; htn, chronic and gestational hypertension.

In addition to exploring the contribution of individual features to the model’s prediction, Shapley values can highlight the interactions between different variables (appendix p 31). For example, the presence of proteinuria is unlikely to be associated with a diagnosis of preeclampsia if the maximal systolic blood pressure during pregnancy is <140 mmHg; however, above that threshold, proteinuria becomes highly predictive. The relationships between some variables follow non-linear patterns; for example, age is a risk factor if the patient is <20 years or >35 years old, especially if associated with higher parity or higher gravidity. A similar J-shaped relationship is also present between age and short or long interpregnancy intervals. Interestingly, higher SBP is likely to be associated with a higher risk of preeclampsia at age <35 years old. A higher red blood cell count during the second trimester is associated with a higher risk for preeclampsia.

Comparison to the current standard of care

Although there is no universally accepted model for the prediction of preeclampsia, the ACOG risk stratification criteria are widely accepted in clinical practice¹. The preeclampsia xgboost model outperformed a model created using the ACOG risk factors based on AUC across all time points (Fig 4A). Moreover, as ACOG guidelines recommend aspirin prophylaxis for all patients at high risk for preeclampsia before 16 weeks, we evaluated how many additional patients would have benefitted from aspirin who would not have been identified by ACOG risk criteria. We compared the performance of our xgboost model with ACOG risk criteria in the test dataset. Using our xgboost model predictions at 14 weeks on the test dataset, 25.0% (N=5624) individuals would be eligible for aspirin prophylaxis, while only 14.6% (N=3295) would be eligible using ACOG criteria. Using the novel preeclampsia model, an additional 2,329 individuals would have been eligible. If aspirin prophylaxis prevents 62% of early-onset preeclampsia in high-risk patients,²³ an additional 28 cases (total of 66 cases) of early-onset preeclampsia per 10,000 pregnant individuals would have the potential to be prevented with updated risk prediction.

• Download figure
• Open in new tab

Fig 4.

Comparison between the xgboost model and the current standard of care as determined by ACOG guidelines. A. Comparison between AUC of the xgboost model and the ACOG criteria over all time timepoints. B. Venn diagram of all individuals at risk for preeclampsia in the test dataset (N=22,511); of those, individuals at risk predicted using the ACOG criteria (acog) are in pink, and the individuals at risk predicted using the xgboost model (model) at 14 weeks are in green.

DISCUSSION

We developed an accurate artificial intelligence (AI) tool consisting of eight longitudinal models for preeclampsia prediction using machine learning. With the advancement of pregnancy and the increase in the gestational weeks, the models get progressively more accurate, from area under the curve (AUC) 0.76 to 0.91; the highest predictive value, AUC 0.91, was achieved before delivery. The most predictive features were systolic and diastolic blood pressure and interpregnancy interval. As gestational age advanced, the relative significance of systolic and diastolic blood pressure compared to other predictors increased significantly. Compared with the current standard of care, our models have higher predictive power and would allow earlier and more precise identification of patients at risk for preeclampsia.

The importance of screening for preeclampsia throughout pregnancy has been emphasized,¹ and to date, multiple predictive models have been developed.^{9–11,16–19} Most studies include clinical data as well as biomarkers, such as serum placental growth factor,^8,19 pregnancy-associated plasma protein A,^8,19 and uterine artery pulsatility index.^6,8,19 However, these tests are not routinely performed in all clinical settings, and thus, the models that utilize these data can be applied only in limited cases. We aimed to develop an accurate tool that can be applied throughout pregnancy and utilizes EHR data routinely collected during prenatal visits. In this way, our AI system has the advantage of being developed using real-world clinical data and can be reproduced in any setting that utilizes EHR.

We achieved high predictive power using our models, which are comparable to or better than other studies using EHR.^6,10,11,18 We developed multiple models and demonstrated that preeclampsia can be predicted with high performance. The model with highest AUC was xgboost; this type of model has a low risk of overfitting, high interpretability, and performs well in clinical structured data.²⁴ In addition, this type of model was highly performant in prior studies predicting preeclampsia.^10,11 However, these prior studies developed models from all available EHR data, which may lead to overfitting, ^10,18 did not investigate multiple machine learning models, ^10,11 and the datasets have high missingness.¹¹ In addition, some prior results may display data leakage, for example, including oxytocin, an intra- or post-partum medication in an early antepartum model. We overcome the limitations of prior work by including only those variables which are clinically relevant for the diagnosis of preeclampsia in our models; we anticipate that this approach will improve cross-institutional reproducibility. The external validation and bias investigation are discussed in detail in the Supplemental Discussion (appendix p 32).

The current study is the largest investigation to date based on the cohort size, time points, depth of predictor variables, and types of machine learning models. We demonstrate the strong effect on preeclampsia risk of known predictors, such as a past medical history of preeclampsia, diabetes, and chronic kidney disease.^7,11,16 In addition, we also incorporated time-series variables in our models. This approach improves the accuracy of predictions,^9,10 and we replicate prior findings of heightened preeclampsia risk with excessive weight gain^9,25,26 and steep patterns of blood pressure increase^9,10,27 during pregnancy. As evidence for the relative role of individual risk factors and interaction between those risk factors is lacking,²⁶ our study adds substantial results from clinical practice to inform future predictive tools and guidelines.

The advantage of using Shapley values as an explainable AI approach is that we demonstrate the relative significance of individual factors and the temporal change as the gestational age advances. Not surprisingly, the most important features in early pregnancy are chronic hypertension and interpregnancy interval, which are part of the medical history. Later in pregnancy, the trajectory of the systolic and diastolic blood pressure becomes more important, which potentially may present a subclinical presentation of developing disease. While individual and groups of risk factors have been well investigated,²⁶ only a few prior studies have analyzed all patient EHR data during pregnancy in its entirety.^9,10 Our results offer a more granular look at the individual variables and demonstrate a higher significance of the patient history in early pregnancy, compared to other studies that rank higher vital signs and laboratory results.¹⁰ These differences likely represent variable institutional practices for early pregnancy screening and highlight the need for future large, multi-center studies.

Preeclampsia risk factors can interact in a complex, non-linear fashion. For example, chronic hypertension is associated with early-onset preeclampsia in patients with weight < 69 kg, no diabetes, and no family history of preeclampsia.⁷ When analyzing the relationships between variables in our models, we demonstrate previously known, as well as new, interactions. The risk of preeclampsia is higher in the extremes of age,²⁸ and also in pregnancies with short and long interpregnancy intervals.²⁹ The connection with red blood cell count is novel and may be related to a higher hemoconcentration.³⁰ As preeclampsia is a heterogeneous disorder, further exploring the interaction between variables has the potential to identify sub-phenotypes of preeclampsia that may benefit from a more personalized approach to prophylaxis and treatment.³¹

Our ability to utilize the Shapley method on an individual level highlights the clinical value of these predictive models to personalize patient management. Identifying the risk factors will allow visualization and potential modification of those factors, which can change the course of the disease and decrease risk. If implemented longitudinally during the pregnancy, for example, in the form of a dashboard, these individualized predictors would allow earlier detection and amelioration in high-risk patients or adjustment of surveillance in low-risk individuals. While individual Shapley values have inherent limitations, this approach can be used to evaluate bias and overall performance.³² Across the population, this strategy would allow efficiency and cost savings while maintaining high standards of care.

There is no definitive therapy for preeclampsia (other than delivery); therefore, efforts are focused on prevention and early detection. Multiple studies have shown that aspirin prophylaxis started in early pregnancy in patients at high risk can decrease the incidence of early-onset preeclampsia, which is associated with the highest incidence of maternal and neonatal morbidity.^23,33 As high-risk patients benefit from aspirin prophylaxis, accurate risk stratification is paramount. The current ACOG guidelines recommend prophylaxis and increased surveillance of high-risk patients;¹ however, multiple studies have demonstrated that the predictive power of the current standard of care using established risk factors is limited. ^10,11,26 Therefore, implementing our AI tool in clinical practice may offer substantial benefits for patient health.

Our study has several limitations. In some cases, the diagnosis of preeclampsia may be challenging to establish as there may be no clear tests that distinguish between gestational hypertension, chronic hypertension, and superimposed preeclampsia. There is also a degree of uncertainty in the diagnostic criteria definitions. ^1,34 Traditionally, most retrospective and modeling studies have used disease codes,⁹ which may be inaccurate.³⁵ To improve the accuracy of the outcome, we developed and manually validated an algorithm based on structured data. In addition, as some patients utilized home blood pressure measurements, those values were not available in the EHR and, thus, were not included in the models. In the future, integrating home blood pressure monitoring may improve the predictive power of models. We also acknowledge that ACOG risk factors¹ may not be an ideal comparative model to our models; however, this approach has been used by others,¹⁰ and most modeling studies have insufficient data to reproduce the results.¹⁷ Lastly, we used retrospective data to test and externally validate our models, and further prospective validation is needed to evaluate the true predictive power.

By using routinely collected data during scheduled prenatal visits, we demonstrate that accurate prediction of preeclampsia can be achieved. This design would allow risk assessment utilizing these models at every visit throughout pregnancy with updates in risk prediction, resulting in a more accurate ascertainment of individuals at risk who would benefit from prophylactic measures or increased surveillance. In the future, as better testing or prophylaxis become available, these AI tools can be used to select the group of individuals who would benefit the most.

Contributions

B.E., V.K., and K.G. conceptualized and designed the study. B.E., R.C., and V.K. conducted all analyses, J.R. assisted with data processing. B.E. and V.K. drafted the manuscript, which was edited and critically reviewed by R.C., J.R., D.B., and K.G. Mass General Brigham was the source of data and location for all analyses. All authors had access to all study data. All authors read and approved the final manuscript and had final responsibility for the decision to submit it for publication.

Competing interests

KJG has served as a consultant to Illumina Inc., Aetion, Roche, and BillionToOne outside the scope of the submitted work. DWB reports grants and personal fees from EarlySense, personal fees from CDI Negev, equity from Valera Health, equity from CLEW, equity from MDClone, personal fees and equity from AESOP Technology, personal fees and equity from FeelBetter, and grants from IBM Watson Health, outside the submitted work. VPK reports consulting fees from Avania CRO unrelated to the current work.