ABSTRACT
Objective Risk stratification in paediatric patients undergoing heart surgery remains a challenge. The improving partial risk adjustment in surgery (PRAIS2) is a risk model predicting 30-day mortality which has been recently developed and validated using a UK-based cohort from April 2009-March 2015. We aimed to perform an independent temporal external validation to explore its generalisability and clinical utility.
Methods PRAIS2 validation was carried out using a single centre (Bristol, UK) cohort from April 2004 to March 2009 and April 2015 to July 2019. For each subject PRAIS2 score was calculated according to the original formula. PRAIS2 performance was assessed in terms of discrimination by means of ROC curve analysis and calibration by using the calibration belt method.
Results A total of 1330 (2004-2009) and 1187 (2015-2019) paediatric cardiac surgical procedures were included in the first and second independent validation, respectively (median age at the procedure 6.0 and 6.9 months). PRAIS2 score showed excellent discrimination for both independent validations (AUC 0.72 (95%CI: 0.65 to 0.80) and 0.87 (95%CI: 0.82 to 0.93), respectively). While PRAIS2 was only marginally calibrated in the first validation, with a tendency to underestimate risk P-value = 0.051), the second validation showed good calibration with 95% confidence belt containing the bisector for predicted mortality (P-value = 0.15); We also observed good performance in the subgroup of patients undergoing non-elective procedures (N = 482; AUC 0.78 (95%CI 0.68 to 0.87); Calibration belt containing the bisector (P-value=0.61).
Conclusions In a single centre UK-based cohort, PRAIS2 showed excellent discrimination and calibration in predicting 30-day mortality in paediatric cardiac surgery including in those undergoing non-elective procedures. Our results support a wider adoption of PRAIS2 score in the clinical practice.
Strengths and limitations of this study
A strength of the present study is that data were prospectively collected as part of the UK National Congenital Heart Disease Audit and as such they undergo continuous and inclusive systematic validation that includes the review of a sample of case notes by external auditors to ensure coding accuracy.
We used a recently proposed method (calibration belt) which does not require patients to be categorised according to risk percentile but rather provides a risk function across all risk value with relative uncertainty measure (95% CI)
A key limitation of this study is that the sample size is relatively small and considerably smaller than the cohort used to develop PRAIS2
Key questions
What is already known about this subject? The improving partial risk adjustment in surgery (PRAIS2) is a risk model predicting 30-day mortality which has been recently developed and validated using a UK-wide cohort.
What does this study add? The present study reported the first independent external validation of the PRAIS2 using a single centre cohort which confirmed excellent performance of the model and for the first time showed that it also accurately predicts mortality in patients undergoing non-elective procedures
How might this impact on clinical practice? Our results support a wider adoption of the PRAIS2 in the clinical practice.
Competing Interest Statement
DAL has received support from several national and international charity and government grants and from Medtronic Ltd and Roche Diagnostics for research unrelated to that presented here. The other authors declare no conflicts of interest.
Funding Statement
This study was supported by the British Heart Foundation Accelerator Award (AA/18/7/34219) and the Bristol National Institute of Health Research Biomedical Research Centre. LC, RC, and DAL work in a unit that receives support from the University of Bristol and the UK Medical Research Council (MC_UU_00011/6). DAL is a National Institute of Health Research Senior Investigator (NF-0616-10102). The views expressed in this publication are those of the author(s) and not necessarily those of the UK National Health Service, the National Institute for Health Research or the UK Department of Health and Social Care, or any other funders mentioned here.
Author Declarations
All relevant ethical guidelines have been followed; any necessary IRB and/or ethics committee approvals have been obtained and details of the IRB/oversight body are included in the manuscript.
Yes
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
M.Caputo{at}bristol.ac.uk, rosie.cornish{at}bristol.ac.uk, d.a.lawlor{at}bristol.ac.uk
Data Availability
The authors confirm that the data supporting the findings of this study are available within the article or its supplementary materials.