Is it time to use machine learning survival algorithms for survival and risk factors prediction instead of Cox proportional hazard regression? A comparative population-based study =================================================================================================================================================================================== * Sara Morsy * Truong Hong Hieu * Abdelrahman M Makram * Osama Gamal Hassan * Nguyen Tran Minh Duc * Ahmad Helmy Zayan * Le-Dong Nhat-Nam * Nguyen Tien Huy ## Abstract **Purpose** Applying machine learning in medical statistics offers more accurate prediction models. In this paper, we aimed to compare the performance of the Cox Proportional Hazard model (CPH), Classification and Regression Trees (CART), and Random Survival Forest (RSF) in short-, and long-term prediction in glioblastoma patients. **Methods** We extracted glioblastoma cancer data from the Surveillance, Epidemiology, and End Results database (SEER). We used the CPH, CART, and RSF for the prediction of 1- to 10-year survival probabilities. The Brier Score for each duration was calculated, and the model with the least score was considered the most accurate. **Results** The cohort included 26473 glioblastoma patients divided into two groups: training (n = 18538) and validation set (n = 7935). The average survival duration was seven months. For the short- and long-term predictions, RSF was the best algorithm followed by CPH and CART. **Conclusion** For big data, RSF was found to have the highest accuracy and best performance. Using the accurate statistical model for survival prediction and prognostic factors determination will help the care of cancer patients. However, more developments of the R packages are needed to allow more illustrations of the effect of each covariate on the survival probability. Keywords * Random survival forests * CART * Cox proportional hazard model * Big data * Artificial intelligence * Glioblastoma * Survival * brier score ## Introduction Malignant brain tumors are among the most formidable types of cancer, with their poor prognosis and the direct influence on cognitive functions, working ability, quality of life [1]. In 2010, according to the prevalence estimate, nearly 200000 patients were diagnosed with primary malignant brain tumors in the United States [2]. Among all the primary malignant brain tumors, malignant gliomas are the most common type with 80% of patients and an annual incidence of 5.26 per 100 000 population, which also means 17000 new cases diagnosed per year [3]. This disease, however rare in children, may present at any age, but it peaks in the sixth through eighth decades of life. Moreover, the number of patients is expected to increase with the aging of the population [3-5]. Glioblastoma (GB) is the most frequent subtype that comprises 51% of all gliomas [6]. Because of its location in the brain, aggressiveness, and low survival duration, GB is well-known as one of the most lethal cancers [7]. The common symptoms in the last month of life include seizures, headache, drowsiness, dysphagia, and eventually death rattle, agitation, and delirium. In the last stage of the disease, patients need appropriate palliative care to allow them to experience a peaceful death despite their severe symptoms [7]. According to 2016 CNS WHO classification, glioblastomas are separated into glioblastoma, IDH-wildtype (presenting in 90 % of cases), which conforms most frequently with the clinically defined primary and prevails in patients over 55 years of age [1, 8]; glioblastoma, IDH-mutant (in about 10 % of cases), which conforms closely to secondary glioblastoma and tendentiously appears in younger patients [8]; and another is glioblastoma, NOS (not otherwise specified), a diagnosis for those tumors which full IDH evaluation cannot be performed [1]. In research, especially epidemiological topics, scientists often encounter multilevel or hierarchical data, such as evaluating the potential characteristics of patients, hospitals, and regions related to the risk of death in those patients with glioblastoma during a specified duration. Survival analysis, thus, refers to methods for the analysis of data in which the outcome demonstrates the time to the onset or occurrence of a targeted event. This method of analysis has the characteristic of censorship: the event may not occur for all subjects before the completion of the study and, at the end of the study, those event-free subjects are said to be censored [9, 10]. The most basic method for survival analysis is survival tables [9, 10]. The time is divided into intervals. For each interval, the count and proportion of each living, censored and death cases are calculated. The most widely used method is the Cox Proportional Hazard regression model or approach (CPH). It estimates the magnitude of the risk of death and its confidence interval. It is used for multiple analyses of survival time data. It is considered a multiple linear regression analysis. CPH analysis depends on the assumption of the proportionality of survival time data [10]. The results are hazard ratios that estimate the probability of an event at a specific type. One of the most common cons of CPH is the convergence or the divergence of the model. This occurs if the assumption of proportionality is not fulfilled or in the presence of many covariables that are not important. It is also reported that CPH analysis had overfitting problems, which means that it describes the random error instead of examining the relationship between the variables [10]. With the development of different machine learning algorithms, new algorithms were used to deal with different limitations and problems in biomedical research. In survival analysis, many algorithms have been used. One of them is recursive partitioning algorithms. These algorithms include Random Survival Forests (RSF) and Classification and Regression Trees (CART) [11-13]. The CART algorithms have gained popularity over time due to easy implementation and interpretability. It is based upon using important variables to split the data into many nodes representing the predictors. Impurity parameters are used to identify whether to select the splitting variable and whether to continue or stop the splitting. In each node, there is a strong association between splitting variables and the response variables evidenced by the highest impurity reduction [14, 15]. Meanwhile, an RSF is an algorithm of an ensemble of trees; it is the average prediction of all trees that would produce a more accurate prediction. It is a robust algorithm against overfitting and resistant to outliers and high dimensionality data. It is a nonparametric method that can be used on any variables regardless of the distribution they follow [12, 13, 15]. In this study, we aim to compare the performance of the Cox Proportional Hazard approach (CPH), CART, and Random Survival Forest (RSF) in short-, and long-term prediction in glioblastoma patients. ## Methods ### Data collection We extracted the data of cases diagnosed with glioblastoma as reported in the histology recode of brain grouping in the SEER database. We used the last version of the published US research data (1975 – 2018) released on April 2021 [16]. This included glioblastoma, NOS (9440/3), Gliosarcoma (9442/3), and giant cell glioblastoma (9441/3). SEER*stat 8.3.9.2 was used to extract the data [17]. We only included cases that died due to the tumor itself and survival time more than zero. Cases with a survival time of zero were excluded. Based on literature, long-term survival was defined in literature as survival of glioblastoma patients more than two years [18]. ### Statistical analysis The categorical variables were expressed in percentage while the continuous variables are expressed using mean and standard deviation if the data are normally distributed; otherwise, median, and interquartile range. The dataset was divided randomly into training groups with 70% of the cases and a test set with 30% of the dataset using the caret package. In the training set, missing data were imputed using the K nearest neighbors (KNN) algorithm with the number of neighbors equal to 3. The imputation was conducted in R using the VIM package [19]. For the validation set, we did sensitivity analysis where we compared the performance results between * i. Imputation of the missing data of the validation set using KNN * ii. Omitting the missing data from the validation set Three survival algorithms were performed and compared based on an Brier Score (BS) for survival analysis on intervals of 12 months to compare the performance of the models for short- and long-term prediction. Univariable Cox Proportional Hazard regression analysis (CPH) was first developed to detect the significant covariables which were used in the multivariable analysis. Then, the accuracy of the model was assessed on the validation set using Brier score. Random Survival Forests (RSF) were applied using the randomforestSRC package in R [20, 21]. We used 500 trees with three variables to split at each node. The variable importance was detected using the permutation method. The predictive performance of the random forest was assessed using brier score on the validation data. The p-value for significant important variables was calculated based on Altmann et al. that depends on permutation importance [22]. A CART survival decision tree was constructed using the Rpart package; we used the minimum variable at each split of 10 and a maximum depth of 10 then we pruned the tree to avoid overfitting. Results were considered significant when the *p-value* was less than 0.05. ### Brier score In this paper, the Brier score was used to compare the accuracy of prediction of each model. The Brier score measures the accuracy of the prediction. Brier score is an evaluation metric that calculates the weighted average of squared error between the event status at time t and predicted survival probability at time t. The higher the value of the Brier score, the less accurate the results are [23, 24]. The scores were calculated using the ipred package [25] ## Results ### Patients’ characteristics The cohort included 26473 glioblastoma patients including training (n = 18538) and validation set (n = 7935). The median age of patients was 64 years old (Table 1). White males had the highest rates of glioblastoma. The most common site for glioblastoma was the frontal lobe. The median survival time for all patients was eight months. The cohort included patients who had anaplastic undifferentiated tumors (n = 7370). 77.4% of patients received surgical treatment; 93.4% of patients did not survive. View this table: [Table 1.](http://medrxiv.org/content/early/2021/11/21/2021.11.20.21266627/T1) Table 1. Characteristics of the included patients ### Survival analysis using Cox Proportional Hazard regression (CPH) The univariable analysis revealed that all patients characteristics except gender of the patients had increased or decreased the mortality rate, for instance, older age patients had low survival probability [HR = 1.2, 95% CI (0.87, 1.5), p-value <0.001]. For tumor characteristics, pathology, different sites of the tumor, Grade, and laterality had significantly affected the survival of glioblastoma patients (Table 2). In the multivariable analysis, old white patients were diagnosed with undifferentiated tumors in the optic nerve with more than one primary tumor (Table 2). Moreover, Gliosarcoma had decreased survival probability (Table 2). View this table: [Table 2.](http://medrxiv.org/content/early/2021/11/21/2021.11.20.21266627/T2) Table 2. Univariable and multivariable cox proportional hazard model The accuracy of the model was checked using the Brier Score (BS) for both short- and long-term predictions. We found that the accuracy of the model was the least in the first year, followed by two years. After the sixth year, the model had the best accuracy that was sustained until 10-year predictions (Table 3). The same was found in the separate analysis with the validation set imputed through KNN. For machine learning algorithms. View this table: [Table 3.](http://medrxiv.org/content/early/2021/11/21/2021.11.20.21266627/T3) Table 3. Comparison between each model using brier score to estimate the accuracy of each model for short- and long-term prediction in test set with or without missing data imputation Ensemble trees were constructed using 500 trees with three variables used at each split. The trees identified age, laterality, and surgery as the top three important variables for the prediction of patient survival (Figure 1). Random forest survival trees had high accuracy (low brier score); the BS was 0.177 for the first year and decreased for the long-term analysis (0.01 for 10 years’ prediction) Table 3. ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/11/21/2021.11.20.21266627/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2021/11/21/2021.11.20.21266627/F1) Figure 1. Variable importance plot illustrating the top variables used for prediction. Site_1 is the specific central nervous system sites. For the CART survival tress that used all the predictors for survival prediction, the cohort was divided into six groups as shown in (Figure 2). The groups are: 1) Patients aged more than 66.5 and performed surgery (median survival = 6 months, n = 5079, 2) Patients aged more than 66.5 and did not perform any surgery (median survival = 3 months, n = 1872), 3) Patients who aged less than 45.5 (median survival = 16 months, n = 2283) have survival probability of 0.538, 4) Patients whose age are between 45.5 – 66.5 and performed surgery (median survival = 11 months, n = 7271), 5) Patients whose age are between 45.5 – 66.5 and did not perform surgery (median survival = 5 months, n = 1578) ![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/11/21/2021.11.20.21266627/F2.medium.gif) [Figure 2.](http://medrxiv.org/content/early/2021/11/21/2021.11.20.21266627/F2) Figure 2. The recursive partitioning survival trees illustrating the survival probability for each group of patients. Assessing the accuracy of the CART model using BS revealed that CART has much less accuracy than CPH and random forest implying that it is not suitable for long-term prediction Table 3. The accuracy of the model is decreasing through time with the highest accuracy in the first year (Figure 3). ![Figure 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/11/21/2021.11.20.21266627/F3.medium.gif) [Figure 3.](http://medrxiv.org/content/early/2021/11/21/2021.11.20.21266627/F3) Figure 3. Boxplot showing the overall median brier score for each model suggesting random forest had the highest accuracy (A), Figure B shows the brier score at each time point indicating that random forest had the lowest brier score hence the highest accuracy Based on our results, we found that random forest maintained high accuracy (low Brier score) for both short- and long-term predictions followed by CPH followed by CART survival trees Figure 3. Brier score for CPH model decreased for longer duration indicating the high accuracy of model for long term prediction, however, Brier score was higher than random forest suggesting random forest is the most suitable model for short- and long-term predictions Table 3. Recursive partitioning trees showed very low accuracy compared to other models. ## Discussion This study aimed at investigating the appropriate survival models to use for the big data in medicine. We compared three models using Brier Scores (BS) as an indicator of model accuracy. Based on our results, RSF had the best accuracy for short- and long-term predictions followed by CPH. Machine learning introduction into medical sciences and data analytics has created a massive impact in the field of public health [26]. With the help of automated processes and artificial intelligence, we acquired the ability to read unrevealed data and algorithm patterns. According to Nasejje et al., when faced with irrelevance between the covariates in the model for time-to-event data, they use the Cox model and link each covariate with one public health assumption [27]. On the other hand, when using RSF, the data will be not reliant on the assumption for its validity [28]. So, Nasejje et al. used the RSF to analyze public health data to figure out associated factors with mortality of younger group of patients (under the age of five) [27]. Typically, in any dataset that includes many covariables carrying the risk of violating the proportional hazards assumption, RSF can be considered [29, 30]. Moreover, Random Survival Forests (RSF), in several risk models, outperformed the traditional Cox Proportional Hazard model (CPH). More importantly, while Cox cannot automatically identify the nonlinear effects of all considered variables, RSF can [28, 30-33]. However, in 2009, on breast cancer patients, the authors reported that CPH was a reliable method for predicting disease-free survival (DFS) in cancer. It was more advantageous than RSF approaches. This was justified by the capability of CPH to extract patterns and relationships hidden deep into medical datasets, leading to high predictive abilities that can be used for different sample sizes and potential future suitable survival data problems, whereas RSF provides only interpretive results [34]. For a dataset with separated and different risk levels, “Model-Based Recursive Partitioning” was able to present a good description. Safe M et al. reported the superiority of recursive partitioning for nonlinear model structures [35]. In the interaction dataset, recursive partitioning like Classification and Regression Trees (CART) and Artificial Neural Network (ANN) showed superior results to Cox (*P*< 0.05) with an improvement of 0.1 (95% CI, 0.08 to 0.12) and 0.015 (95% CI, 0.01 to 0.02) respectively. In theory, CART and ANN overcame the limitations of the Cox model regarding the ease and extent of their use [36]. In a study about breast tumor chemosensitivity to primary chemotherapy, on the other hand, the logistic regression model predictions were better than recursive partitioning [37, 38]. Another study by Lee et al. confirmed that Cox linear regression modeling outperformed recursive partitioning when there were only continuous predictors, while recursive partitioning was better when there were significant categorical predictors [37, 39]. One last study by Chun et al. demonstrated that Artificial Neural Network (ANN) had worse performance than the logistic regression model [37, 40]. Because the main goal of those methods (RSF, CART, and ANN) is to develop a predictive model of many variables that can lead to a more efficient clinical use that is particularly important for physicians. Contrastingly, conventional statistical modeling needs proper input of data from an expert to create a much easier model to interpret than data-driven-based techniques. This, in turn, leads to the narrow scope of conventional techniques to find new correlations between the data ready to be used in the literature [41]. Another study addressing dyslipidemia analyzed the difference in the disease incidence using RSF and Cox model and summarized that the RSF could predict more variables than the CPH. Those variables included the baseline lipoprotein profiles (including high- and low-density lipoproteins, total cholesterol, total triglycerides, blood pressure, age, body mass index, … etc.) [42]. Accordingly, not only do we need a tool to analyze the mortality, morbidity, and risk rates, but also a tool we can depend on searching for more variables effectively. So, using machine learning techniques can help us achieve combinations that we can barely capture using conventional approaches [41]. In our models, we validated our results using Brier Score (BS). An earlier study published in 2009 compared the RSF to CPH, using the Harrell c-statistics [43-45] for assessing the validity, found that CPH had better performance, and concluded that its replacement by RSF is still controversial and needed further investigation [46]. On the other hand, two other studies, assessing the 1-year mortality and survival in cardiac patients with cardiac arrhythmias, compared the use of RSF and the traditional CPH. The results were that RSF significantly overperformed CPH [47, 48]. The latter findings were supported by numerous other published studies [31, 42, 49, 50]. One of the biggest obstacles is not only to collect datasets that have the appropriate size and needed quality of samples but also to use appropriate methods in analysis. Towards that point, our study contributes as it compares these predictive methods which in turn can improve and reinforce the theory about the limitations and advantages of each method [26]. ## Limitation Our analyses were conducted in R. The effect of each level of the categorical variables in the survival trees could not be aptly illustrated due to the lack of R packages with more advanced illustrations. Another limitation is that random forest and the function used to calculate brier score failed if the data has missing values. ## Conclusion In this paper, we compared the performance of the Cox Proportional Hazard model (CPH), Classification and Regression Trees (CART), and the Random Survival Forest (RSF) in predicting the survival of glioblastoma patients reported in the SEER database. We concluded that the RSF achieved the best performance and the highest accuracy followed by the CPH and lastly by the CART for both short- and long-term predictions, validated by the Brier Score (BS). Accordingly, using RSF may be of benefit in determining the best prognostic factors of cancer patients; however, more development of R packages is needed to allow for more illustrations of each covariate effect. More studies of the same kind are also needed to examine the performance of the three models in other cancer types. ## Data Availability Availability of data and material The data are available and can be accessed through the SEER database, which is publicly available at [https://seer.cancer.gov/data/](https://seer.cancer.gov/data/). [https://seer.cancer.gov/](https://seer.cancer.gov/) ## Declarations ### Ethics approval and consent to participate Not applicable ### Consent for publication The authors of this manuscript consent to the publication of the work by BioMed Central. ### Availability of data and material The data are available and can be accessed through the SEER database, which is publicly available at [https://seer.cancer.gov/data/](https://seer.cancer.gov/data/). ### Competing interests None ***Funding*** None ### Authors’ Contributions SM is responsible for the idea of the study, concept and design, acquisition of the data from the SEER database, statistical analysis and interpretation of data. SM, AMM, HT, OGH, NTMD, LDNN, and AHZ contributed to writing the manuscript. The study was under the supervision of NTH, who has also revised the manuscript. All authors read and approved the final version of the manuscript. ## Code availability The codes used in the analysis are available upon request from Dr. Sara Morsy (sara.morsy{at}med.tanta.edu.eg) ## Table legends **Table 1**. The characteristics of included patients. **Table 2**. Univariable and multivariable Cox regression analysis **Table 3**. Comparison between each model using brier score for each model to estimate the accuracy of each model for short- and long-term prediction ## Acknowledgments None ## Footnotes * **SM:** sara.morsy{at}med.tanta.edu.eg **THH:** hhieu.truong{at}gmail.com **AMM:** abd-makram{at}hotmail.com **OGH:** osamagamal4842{at}gmail.com **NTMD:** minhuc1298{at}gmail.com **AHZ:** ahmad_zayan{at}yahoo.com **LDNN:** bacsinam81{at}gmail.com **NTH:**tienhuy{at}nagasaki-u.ac.jp * 3 Online Research Club ([www.onlineresearchclub.org](http://www.onlineresearchclub.org)) * Received November 20, 2021. * Revision received November 20, 2021. * Accepted November 21, 2021. * © 2021, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. 1.Louis DN, Perry A, Reifenberger G, von Deimling A, Figarella-Branger D, Cavenee WK, et al. The 2016 World Health Organization Classification of Tumors of the Central Nervous System: a summary. Acta neuropathologica. 2016;131(6):803–20. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00401-016-1545-1&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=27157931&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 2. 2.Porter KR, McCarthy BJ, Freels S, Kim Y, Davis FG. Prevalence estimates for primary brain tumors in the United States by age, gender, behavior, and histology. Neurooncology. 2010;12(6):520–7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/neuonc/nop066&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20511189&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000278817700002&link_type=ISI) 3. 3.Dolecek TA, Propp JM, Stroup NE, Kruchko C. CBTRUS statistical report: primary brain and central nervous system tumors diagnosed in the United States in 2005-2009. Neurooncology. 2012;14 Suppl 5:1–49. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/neuonc/nor225&link_type=DOI) 4. 4.Wohrer A, Waldhor T, Heinzl H, Hackl M, Feichtinger J, Gruber-Mosenbacher U, et al. The Austrian Brain Tumour Registry: a cooperative way to establish a population-based brain tumour registry. Journal of neuro-oncology. 2009;95(3):401–11. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11060-009-9938-9&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19562257&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 5. 5.Ostrom QT, Gittleman H, Stetson L, Virk SM, Barnholtz-Sloan JS. Epidemiology of gliomas. Cancer treatment and research. 2015;163:1–14. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/978-3-319-12048-5_1&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25468222&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 6. 6.Kleihues P, Burger PC, Scheithauer BW. The new WHO classification of brain tumours. Brain pathology (Zurich, Switzerland). 1993;3(3):255–68. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1750-3639.1993.tb00752.x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=8293185&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1993LP45200006&link_type=ISI) 7. 7.Preusser M, de Ribaupierre S, Wohrer A, Erridge SC, Hegi M, Weller M, et al. Current concepts and management of glioblastoma. Annals of neurology. 2011;70(1):9–21. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/ana.22425&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21786296&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 8. 8.Ohgaki H, Kleihues P. The definition of primary and secondary glioblastoma. Clinical cancer research : an official journal of the American Association for Cancer Research. 2013;19(4):764–72. 9. 9.Austin PC. A Tutorial on Multilevel Survival Analysis: Methods, Models and Applications. International statistical review = Revue internationale de statistique. 2017;85(2):185–203. 10. 10.George B, Seals S, Aban I. Survival analysis and regression models. Journal of nuclear cardiology : official publication of the American Society of Nuclear Cardiology. 2014;21(4):686–94. 11. 11.Bou-Hamad I, Larocque D, Ben-Ameur H. A review of survival trees. Statistics Surveys. 2011;5:44–71. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1214/09-SS047&link_type=DOI) 12. 12.Ishwaran H, Lu M. Random survival forests. Wiley StatsRef: Statistics Reference Online. 2007:1–13. 13. 13.Hothorn T, Bühlmann P, Dudoit S, Molinaro A, Van Der Laan MJ. Survival ensembles. Biostatistics. 2005;7(3):355–73. 14. 14.Barlin JN, Zhou Q, St Clair CM, Iasonos A, Soslow RA, Alektiar KM, et al. Classification and regression tree (CART) analysis of endometrial carcinoma: Seeing the forest for the trees. Gynecol Oncol. 2013;130(3):452–6. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ygyno.2013.06.009&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23774300&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 15. 15.Zhou Y, McArdle JJ. Rationale and Applications of Survival Tree and Survival Ensemble Methods. Psychometrika. 2015;80(3):811–33. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 16. 16.National Cancer Institute D, Surveillance Research Program. Surveillance, Epidemiology, and End Results (SEER) Program ([www.seer.cancer.gov](http://www.seer.cancer.gov)) SEER*Stat Database: Incidence - SEER Research Data, 9 Registries, Nov 2020 Sub (1975-2018) - Linked To County Attributes - Time Dependent (1990-2018) Income/Rurality, 1969-2019 Counties. released April 2021, based on the November 2020 submission. 17. 17.Surveillance Research Program, National Cancer Institute SEER*Stat software (seer.cancer.gov/seerstat) version 8.3.9.2. 18. 18.Poon MTC, Sudlow CLM, Figueroa JD, Brennan PM. Longer-term (≥25A1;2 years) survival in patients with glioblastoma in population-based studies pre- and post-2005: a systematic review and meta-analysis. Scientific Reports. 2020;10(1):11622. 19. 19.Kowarik A, Templ M. Imputation with the R Package VIM. Journal of statistical software. 2016;74(7):1 – 16. 20. 20.Ishwaran H, Kogalur UB. Random survival forests for R. R news. 2007;7(2):25–31. 21. 21.Ishwaran H, Kogalur UB, Blackstone EH, Lauer MS. Random survival forests. The annals of applied statistics. 2008;2(3):841–60. 22. 22.Altmann A, Tolosi L, Sander O, Lengauer T. Permutation importance: a corrected feature importance measure. Bioinformatics. 2010;26(10):1340–7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/bioinformatics/btq134&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20385727&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000277447500009&link_type=ISI) 23. 23.Gerds TA, Cai T, Schumacher M. The performance of risk prediction models. Biometrical journal Biometrische Zeitschrift. 2008;50(4):457–79. 24. 24.Graf E, Schmoor C, Sauerbrei W, Schumacher M. Assessment and comparison of prognostic classification schemes for survival data. Statistics in medicine. 1999;18(17-18):2529–45. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/(SICI)1097-0258(19990915/30)18:17/18<2529::AID-SIM274>3.0.CO;2-5&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=10474158&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000082507800027&link_type=ISI) 25. 25.Hothorn APaT. ipred: Improved Predictors. R package version 0.9-9. 2019. 26. 26.Deo RC. Machine Learning in Medicine. Circulation. 2015;132(20):1920–30. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTQ6ImNpcmN1bGF0aW9uYWhhIjtzOjU6InJlc2lkIjtzOjExOiIxMzIvMjAvMTkyMCI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIxLzExLzIxLzIwMjEuMTEuMjAuMjEyNjY2MjcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 27. 27.Nasejje JB, Mwambi H. Application of random survival forests in understanding the determinants of under-five child mortality in Uganda in the presence of covariates that satisfy the proportional and non-proportional hazards assumption. BMC Res Notes. 2017;10(1):459. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 28. 28.Nasejje JB, Mwambi HG, Achia TN. Understanding the determinants of under-five child mortality in Uganda including the estimation of unobserved household and community effects using both frequentist and Bayesian survival analysis approaches. BMC Public Health. 2015;15:1003. 29. 29.Ehrlinger J. ggRandomForests: Exploring Random Forest Survival. arXiv. 2016. 30. 30.Taylor JM. Random Survival Forests. J Thorac Oncol. 2011;6(12):1974–5. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22088987&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 31. 31.Datema FR, Moya A, Krause P, Back T, Willmes L, Langeveld T, et al. Novel head and neck cancer survival analysis approach: random survival forests versus Cox proportional hazards regression. Head Neck. 2012;34(1):50–8. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/hed.21698&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21322080&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 32. 32.Hamidi O, Poorolajal J, Farhadian M, Tapak L. Identifying Important Risk Factors for Survival in Kidney Graft Failure Patients Using Random Survival Forests. Iranian journal of public health. 2016;45(1):27–33. 33. 33.Hsich E, Gorodeski EZ, Blackstone EH, Ishwaran H, Lauer MS. Identifying important risk factors for survival in patient with systolic heart failure using random survival forests. Circulation Cardiovascular quality and outcomes. 2011;4(1):39–45. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiY2lyY2N2b3EiO3M6NToicmVzaWQiO3M6NjoiNC8xLzM5IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjEvMTEvMjEvMjAyMS4xMS4yMC4yMTI2NjYyNy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 34. 34.Albain KS, Barlow WE, Shak S, Hortobagyi GN, Livingston RB, Yeh IT, et al. Prognostic and predictive value of the 21-gene recurrence score assay in postmenopausal women with node-positive, oestrogen-receptor-positive breast cancer on chemotherapy: a retrospective analysis of a randomised trial. The Lancet Oncology. 2010;11(1):55–65. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1470-2045(09)70314-6&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20005174&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000273874100022&link_type=ISI) 35. 35.Safe M, Faradmal J, Mahjub H. A Comparison between Cure Model and Recursive Partitioning: A Retrospective Cohort Study of Iranian Female with Breast Cancer. Comput Math Methods Med. 2016;2016:9425629-. 36. 36.Kattan MW, Hess KR, Beck JR. Experiments to Determine Whether Recursive Partitioning (CART) or an Artificial Neural Network Overcomes Theoretical Limitations of Cox Proportional Hazards Regression. Computers and Biomedical Research. 1998;31(5):363–73. 37. 37.Ballester M, Oppenheimer A, Mathieu d’Argent E, Touboul C, Antoine J-M, Coutant C, et al. Nomogram to predict pregnancy rate after ICSI–IVF cycle in patients with endometriosis. Human reproduction. 2011;27(2):451–6. 38. 38.Rouzier R, Coutant C, Lesieur B, Mazouni C, Incitti R, Natowicz R, et al. Direct comparison of logistic regression and recursive partitioning to predict chemotherapy response of breast cancer based on clinical pathological variables. Breast cancer research and treatment. 2009;117(2):325–31. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s10549-009-0308-2&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19152025&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000269005500012&link_type=ISI) 39. 39.Lee JW, Um SH, Lee JB, Mun J, Cho H. Scoring and Staging Systems Using Cox Linear Regression Modeling and Recursive Partitioning. Methods Inf Med. 2006;45(01):37–43. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16482368&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000234822900006&link_type=ISI) 40. 40.Chun FKH, Graefen M, Briganti A, Gallina A, Hopp J, Kattan MW, et al. Initial Biopsy Outcome Prediction—Head-to-Head Comparison of a Logistic Regression-Based Nomogram versus Artificial Neural Network. European Urology. 2007;51(5):1236–43. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.eururo.2006.07.021&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16945477&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 41. 41.Steele AJ, Denaxas SC, Shah AD, Hemingway H, Luscombe NM. Machine learning models in electronic health records can outperform conventional survival models for predicting patient mortality in coronary artery disease. PLOS ONE. 2018;13(8):e0202344. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0202344&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=30169498&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 42. 42.Zhang X, Tang F, Ji J, Han W, Lu P. Risk Prediction of Dyslipidemia for Chinese Han Adults Using Random Forest Survival Model. Clin Epidemiol. 2019;11:1047–55. 43. 43.Harrell FE, Jr.., Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical tests. Jama. 1982;247(18):2543–6. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.1982.03320430047030&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=7069920&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1982NP06700024&link_type=ISI) 44. 44.Harrell FE, Jr.., Lee KL, Califf RM, Pryor DB, Rosati RA. Regression modelling strategies for improved prognostic prediction. Statistics in medicine. 1984;3(2):143–52. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/sim.4780030207&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=6463451&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1984SW54000004&link_type=ISI) 45. 45.Harrell FE, Jr.., Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in medicine. 1996;15(4):361–87. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=8668867&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1996TY77400003&link_type=ISI) 46. 46.Kurt Omurlu I, Ture M, Tokatli F. The comparisons of random survival forests and Cox regression analysis with simulation and an application related to breast cancer. Expert Systems with Applications. 2009;36(4):8582–8. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.eswa.2008.10.023&link_type=DOI) 47. 47.Miao F, Cai Y-P, Zhang Y-X, Li Y, Zhang Y-T. Risk Prediction of One-Year Mortality in Patients with Cardiac Arrhythmias Using Random Survival Forest. Comput Math Methods Med. 2015;2015:303250-. 48. 48.Miao F, Cai Y-P, Zhang Y-T, Li C-Y, editors. Is Random Survival Forest an Alternative to Cox Proportional Model on Predicting Cardiovascular Disease? 6th European Conference of the International Federation for Medical and Biological Engineering; 2015 2015//; Cham: Springer International Publishing. 49. 49.Dietrich S, Floegel A, Troll M, Kuhn T, Rathmann W, Peters A, et al. Random Survival Forest in practice: a method for modelling complex metabolomics data in time to event analysis. Int J Epidemiol. 2016;45(5):1406–20. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/ije/dyw145&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=27591264&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F21%2F2021.11.20.21266627.atom) 50. 50.Roshanaei G, Omidi T, Faradmal J, Safari M, Poorolajal J. Determining affected factors on survival of kidney transplant in living donor patients using a random survival forest. Koomesh. 2018;20(3):517–23.