Abstract
Importance Interest in the use of prediction models to support referrals to palliative care is surging. Few high-performing models have been developed, implemented, and pilot-tested following responsible artificial intelligence (AI) principles and transparent reporting guidelines.
Objective Collaborate with physicians and clinical informaticists to plan, develop, validate, implement, and pilot-test a high-performing, statistically fair prediction model—the Serious Illness Clinical Indicator (SICLI)—to support palliative care referrals following responsible AI principles.
Design Cohort study of hospitalizations occurring from January 1st, 2022, to December 31st, 2022.
Setting Kaiser Permanente Southern California
Participants Patients 18+ years, not admitted to maternity or psychiatry departments and hospitalized for at least 36 hours.
Exposures We augmented a commercial prediction model, the Epic End-of-Life-Care Index (EOLCI), by adding locally available predictors, including co-morbidity and laboratory acuity risk scores, nursing flow sheet and health care utilization data.
Main Outcomes and Measures We predicted 12-month mortality risk using a split-sample design, least absolute shrinkage, and selection operator (LASSO) using logistic regression, and five-fold cross-validation. Performance was assessed via the Area Under the Receiver Operating Curve (AUC) and calibration plots. The final model was chosen by clinicians and informaticists for its balance of performance and parsimony. Responsible AI principles guided each development step.
Results Twelve-month mortality rate among 133,043 hospitalizations of at least 36 hours was 23%. Patients who died were older (76±13 vs. 64±17 years) and had higher co-morbidity burden (Charlson, 4.49±3.14 vs. 2.06±2.44). SICLI was very well calibrated and outperformed the EOLCI (AUC of 0.87 (95% CI:0.86, 0.87) vs. 0.81, (95% CI: (0.80, 0.81)). SICLI’s high and very-high risk group (probability cut-off 0.6-<95 and >=0.95), produced positive predictive values (PPVs) of 72.0% and 94.0%, respectively. SICLI was implemented in KPSC’s Electronic Health Record. Post-implementation validation by a palliativist against the 12-month surprise question via review of 25 charts yielded a PPV of 100% and 80% for the very-high and high-risk groups. We report on model documentation, socialization, and governance.
Conclusion and Relevance SICLI is an equitable, high-performing mortality prediction model that builds on and outperforms the EOLCI. It was implemented and pilot-tested in a large health care system.
Question Can we build on a commercially available mortality risk score to develop, implement, and pilot-test a high-performing prediction model to support palliative care decision-making in a large integrated health care system?
Findings The Serious Illness Clinical Indicator (SICLI) augments and outperforms the Epic End-of-Life Care Score (EOLCI) and was implemented and pilot-tested using a responsible AI framework.
Meaning Predictive models can be valuable tools to support palliative care referrals.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was supported by the Kaiser Permanente Southern California Care Improvement Research Team (PIs: Drs. Nau and Nguyen) and the Patient Centered Outcomes Research Institute (PLACER-2022C3-30553, PIs: Drs. Halpern and Courtright).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This study was approved by the Kaiser Permanente Southern California (13886) and University of Pennsylvania (855378) Institutional Review Boards.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Sharing Statement
The data are available for use through collaboration with the KPSC study investigators under the conditions of sufficient funding by the requestor and data use agreements between all institutions that govern data access, storage, and short- and long-term use. Qualified researchers trained in human subject confidentiality protocols interested in collaborating with the KPSC study team can contact Dr. Claudia Nau (claudia.l.nau{at}kp.org).
