Abstract
Background Vitamin B deficiency is common worldwide and may lead to psychiatric symptoms; however, vitamin B deficiency epidemiology in patients with intense psychiatric episode has rarely been examined. Moreover, vitamin deficiency testing is costly and time-consuming. It hampered to effectively rule out vitamin deficiency-induced intense psychiatric symptoms. In this study, we aimed to clarify the epidemiology of these deficiencies and efficiently predict them using machine-learning models from patient characteristics and routine blood test results that can be obtained within one hour.
Methods We reviewed 497 consecutive patients deemed to be at imminent risk of seriously harming themselves or others over 2 years. Machine-learning models were trained to predict each deficiency from age, sex, and 29 routine blood test results.
Results We found that 112 (22.5%), 80 (16.1%), and 72 (14.5%) patients had vitamin B1, vitamin B12, and folate (vitamin B9) deficiency, respectively. Also, the machine-learning models well generalized to predict the deficiency in the future unseen data; areas under the receiver operating characteristic curves for the validation dataset (i.e. dataset not used for training the models) were 0.716, 0.599, and 0.796, respectively. The Gini importance of these vitamins provided further evidence of a relationship between these vitamins and the complete blood count, while also indicating a hitherto rarely considered, potential association between these vitamins and alkaline phosphatase (ALP) or thyroid stimulating hormone (TSH).
Discussion This study demonstrates that machine-learning can efficiently predict some vitamin deficiencies in patients with active psychiatric symptoms, based on the largest cohort to date with intense psychiatric episode. The prediction method may expedite risk stratification and clinical decision-making regarding whether replacement therapy should be prescribed. Further research includes validating its external generalizability in other clinical situations and clarify whether interventions based on this method can improve patient care and cost-effectiveness.
1. Introduction
Vitamin B deficiency is common worldwide and may lead to psychiatric symptoms1–4. For example, meta-analyses have shown that patients with schizophrenia or first-episode psychosis have lower folate (vitamin B9) levels than their healthy counterparts4,5. Moreover, vitamin therapy can effectively alleviate symptoms in a subgroup of patients with schizophrenia3,6–8. However, the epidemiology of vitamin B deficiency in patients with active mental symptoms requiring immediate hospitalization has rarely been examined.
In a psychiatric emergency, psychiatrists should promptly distinguish treatable patients with altered mental status due to a physical disease from patients with an authentic mental disorder (international statistical classification of diseases and related health problems-10, ICD-10 code: F2-9). However, vitamin deficiency testing is very costly (around 60 dollars for each measurement of vitamin B1 (vitB1), vitamin B12 (vitB12), or folate in the U.S.; 15–25 dollars for each test in Japan) and usually requires at least two days. Therefore, an efficient, cost-effective method of predicting vitamin B deficiency is needed.
Although several studies have applied machine-learning to the prediction of diagnosis or treatment outcomes9–11, no study using machine-learning has focused on vitamin B deficiencies. We herein explore whether vitB1, vitB12, and folate deficiencies can be predicted using a machine-learning classifier from patient characteristics and routine blood test results obtained within one hour based on a large cohort of patients requiring urgent psychiatric hospitalization.
2. Methods
2.1. Medical chart review
We reviewed consecutive patients admitted to the Department of Neuropsychiatry at Tokyo Metropolitan Tama Medical Center between September 2015 and August 2017 under the urgent involuntary hospitalization law, which requires the immediate psychiatric hospitalization of patients at imminent risk of seriously harming themselves or others. The necessity of hospitalization was judged by designated mental health specialists. The patient characteristics, ICD-10 codes, and laboratory data were gathered retrospectively.
Since the reference ranges for vitB1, vitB12, and folate are 70–180 nmol/L (30–77 ng/mL), 180–914 ng/L, and > 4.0 μg/L, respectively12, a deficiency of the nutrients was defined as < 30 ng/mL, < 180 ng/L, and < 4.0 μg/L, respectively, unless otherwise stated.
2.2. Random forest classifier and statistics
A random forest classifier was trained to predict the deficiency of each substance from age, sex, and 29 routine blood variables (described in the Result section with values). The random forest classifier was trained using the dataset populated in the period from September 2015 to December 2016 (the “Training set”). First, we optimized the hyperparameters of the classifier by selecting the best combination of hyperparameters that maximized the “5-fold cross validation” accuracy, among many combinations within appropriate ranges. The cross-validation accuracy was computed as follows; in one session, the classifier was trained using 80% of the training set and evaluated on the withheld 20% of the training set. This session was performed five times so that every data would be withheld once. The accuracies were finally averaged across sessions to yield the cross-validation accuracy. By incorporating this process, the classifier was generalized to unseen data (Graphical method is shown in Figure 1).
Using the optimized hyperparameters, the classifiers were then validated using data collected from January 2017 through August 2017 (the “Validation set”). We report the classification performance on the validation set in the results section unless otherwise stated. We quantified the sensitivity, specificity, and accuracy (defined as the average of the sensitivity and the specificity on the optimal operating point) using receiver operating characteristic curves (ROCs). We also quantified the 95% confidence interval of the accuracy using 1000-times bootstrapping.
When investigating the Gini importance and the partial dependency13, we retrained the classifiers using all datasets. All data analyses were performed using Python (2.7.10) with the Scikit-learn package (0.19.0) and R (3.4.2) with the edarf package (1.1.1).
2.3. Ethical considerations
Informed consent was obtained from participants using an opt-out form on the website. The study protocol was approved by the Research Ethics Committee, Tokyo Metropolitan Tama Medical Center (Approval number: 28-8). The study complied with the Declaration of Helsinki and the STROBE statement.
3. Results
3.1. Eligible patients
During the 2-year study period, 497 consecutive patients (496 were Asian) were enrolled. The mean age (standard deviation, SD) was 42.3 (±15.4) years, and 228 patients (45.9%) were women. F2 (Schizophrenia, schizotypal, delusional, and other non-mood psychotic disorders) was diagnosed in over 60% of the patients. The ICD-10 codes of the patients and the number of deficiencies at several cut-off values for vitB1, vitB12, and folate are shown in Table 1. According to the predefined cut-off values12, 112 (22.5%), 80 (16.1%), and 72 (14.5%) patients exhibited a deficiency of vitB1 (<30 ng/mL), vitB12 (<180 ng/L), and folate (<4.0 μg/L), respectively. Vitamin B deficiencies in sub-groups are shown in Table 2. A summary of the full dataset is shown in Table 3. Detailed information (sub-datasets) is shown in Supplementary Table 1, 2, and 3 online. Histograms of vitB1, vitB12, and folate values are shown in Figure 2 A-C.
3.2. Prediction via machine-learning using routine blood test results
A random forest classifier was trained to predict the deficiency of each substance from patient characteristics and routine blood test results. The classifier was trained using the dataset gathered in the period from September 2015 to December 2016 (the “Training set”, n = 373), which was then validated from January 2017 through August 2017 (the “Validation set”, n = 124).
The area under the ROCs (AUCs) for the validation set were 0.716, 0.599, and 0.796, for vitB1, vitB12, and folate, respectively (Figure 2 D-F and Table 4). With some operative points on the ROC, the sensitivity, specificity and accuracy for the validation set were calculated (Table 4. See also Supplementary Table 4 for training set and Supplementary Table 5 for different operating points).
When the prediction performances were compared between the classifiers trained using the dataset from the F2 population and the classifiers trained using the dataset from the other population, the AUC was not statistically different (DeLong’s test), except in the case of vitB1 (see Supplementary Table 6).
Figure 3 shows the Gini importance (a–c) and partial dependency plots (d–f) for the eight most important variables for each substance. The results provided further evidence of a relationship between the vitamin B levels and complete blood count while also indicating the hitherto rarely considered, potential association between these vitamins and alkaline phosphatase (ALP) or thyroid stimulating hormone (TSH).
3.3. Robustness verification
We verified the robustness of the results by two independent means. First, we used different cut-off values to define the deficiency14–16. Although the AUC for the validation set, shown in Supplementary Table 7, tended to be higher when strict cut-off values were used, the obtained AUCs were not statistically significant (p > 0.05, DeLong’s test with Bonferroni correction).
Second, we trained and evaluated random forest classifiers using a dataset split in a different way; the classifier was trained using the dataset collected in the period from the 31st of January, 2016 to August 2017, which was then validated with data gathered from September 2015 to the 31st of January, 2016. Note that the sample sizes of the training and validation sets were equal to those in the original setting. The AUCs for the validation set were 0.771, 0.621, and 0.745 for vitB1, vitB12, and folate, respectively; none were statistically different from the AUC trained using the original setting (DeLong’s test), further demonstrating the robustness of the performance.
4. Discussion
4.1. Relevance of the present study
Based on the largest cohort to date of patients at imminent risk of seriously harming themselves or others, this study indicated that deficiency of certain vitamins can be predicted in an efficient manner via machine-learning using routine blood test results. Given the large number of patients with vitamin B deficiencies, empirical therapy might be acceptable; however, risk stratification is preferred for personalized medicine and shared decision-making. The prediction method presented here may expedite clinical decision-making as to whether vitamins should be prescribed to a patient (Graphical abstract is shown in Figure 4).
Remarkably, the AUC for folate deficiency was 0.796. Folate features the potential to maintain neuronal integrity and is one of the homocysteine-reducing B-vitamins5; homocysteine has been linked to the etiology of schizophrenia17, and vitamin B supplements have been reported to reduce psychiatric symptoms significantly in patients with schizophrenia7. As our study does not present longitudinal results, an intervention effect of folate supplementation in the cohort remains to be clarified.
4.2. Trade-off of interpretability and generalizability using machine-learning
Compared to the AUC of folate, AUCs of vitB1 and vitB12 were relatively low. Using other parameters that were not incorporated into this model or using other models including deep neural networks might increase the accuracy of prediction.
However, interpretability and completeness of machine-learning classifiers are subject to trade-off17. Although completeness and generalizability are desirable, interpretability is also indispensable, especially in clinical settings, since it provides meaningful and trustworthy findings for clinical physicians as well as new biological insights18. In this study we chose random forest classifiers since they provide expressive and interpretable data, with sufficient accuracy.
4.3. Biological mechanism suggestion
Using the random forest classifiers, as shown in Figure 3, we identified several items related to complete blood count as top hits. Notably, our classifier was blind to any biological knowledge, including the well-established association between anemia and vitamin B deficiency, including folate19. The results provide further evidence of a relationship between vitamin B levels and the complete blood count and support the use of machine-learning to investigate novel, underlying biological mechanisms20.
ALP and its metabolites indicate the vitamin B6 status21; low vitB12 is potentially associated with low ALP22. More generally, ALP may have a close and complicated relationship with the overall vitamin B group. Autoimmune disorders, especially thyroid disease, are commonly associated with pernicious anaemia23, but there has been no established hypothesis regarding the causal relationships between thyroid disease and vitamin B deficiencies. The potential association between the levels of these vitamins and ALP or TSH awaits further study, both investigations of populations and basic research24.
4.4. Limitations
This study is subject to several limitations. First, the findings of this single-center retrospective study may have limited generalizability. Second, the patients’ long-term prognosis was not investigated due to administrative restrictions; the extent to which this method can expedite clinical decision-making is therefore unclear. Further, we did not investigate the relationship between serological values and the need for intervention. The lack of data for vitamin B deficiency in the Japanese general population hampered the comparison between the experimental cohort and their counterparts who lacked psychiatric symptoms. Establishing appropriate reference values and an assessment method requires further investigation. Finally, we did not assess the predictive value of other nutritional impairments, including vitamin B6 and homocysteine deficiency, which were previously shown to have a close link with psychiatric symptoms3,5; however, our study provides fundamental data on nutritional impairment based on the largest cohort of patients with intense psychiatric episode ever assembled for this purpose and presents a potential framework for predicting nutritional impairment using machine-learning.
4.5. Conclusion
The present report is, to the best of our knowledge, the first to demonstrate that machine-learning can efficiently predict nutritional impairment. Further research is needed to validate the external generalizability of the findings in other clinical situations and clarify whether interventions based on this method can improve patient care and cost-effectiveness.
Data Availability
The datasets and source code utilized in the current study are available from the corresponding author upon reasonable request.
5. Contribution to the Field Statement
Vitamin B deficiency is common worldwide and may lead to psychiatric symptoms; however, vitamin B deficiency epidemiology in patients with intense psychiatric symptoms has rarely been examined. Moreover, vitamin deficiency testing is costly and time-consuming. Based on the largest cohort to date of patients at imminent risk of seriously harming themselves or others, this study demonstrated that the deficiency of certain vitamins can be predicted in an efficient manner via machine-learning models from patient characteristics and routine blood test results obtained within one hour.
In detail, among the 497 patients investigated (over 60% was diagnosed with schizophrenia or related psychotic disorders), 22.5%, 16.1%, and 14.5% patients had a deficiency of vitamin B1, B12, and folate, respectively, by direct measurement. Also, the machine-learning models well generalized to predict the deficiency in unseen datasets; areas under the receiver operating characteristic curves for the validation dataset were 0.716, 0.599, and 0.796, respectively. The prediction method presented in this study may expedite risk stratification and clinical decision-making regarding whether replacement therapy should be prescribed. The results also provided further evidence for a well-known relationship between these vitamins and the complete blood count and supported the application of machine-learning to investigate novel, underlying biological mechanisms.
7. Author Contributions Statement
H. Tamune has full access to all data and takes responsibility for the integrity of the data. H. Tamune, JU, KN, and NY conceived the study. H. Tamune, YH, and H. Tanaka collected the data. JU performed the statistical analyses. H. Tamune and JU drafted the first version of the manuscript. All authors critically revised the manuscript for intellectual content and approved the final version.
8. Data Availability Statements
The datasets and source code utilized in the current study are available from the corresponding author upon reasonable request.
9. Conflict of Interest Statement
The authors declare no conflict of interest, except for a scholarship grant awarded to JU from Takeda Science Foundation and Masayoshi Son Foundation.
Supporting Material List
Supplementary Table 1 (related to Table 1). Divided patient distribution data (n = 497)
Supplementary Table 2 (related to Table 2). Divided data of vitamin B deficiencies in sub-groups
Supplementary Table 3 (related to Table 3). Divided dataset of age, sex, and 29 parameters
Supplementary Table 4 (related to Table 4). Summary of sensitivity, specificity, and accuracy for the training set
Supplementary Table 5 (related to Table 4). Sensitivities and specificities at other operating points
Supplementary Table 6. Subgroup analyses
Supplementary Table 7. AUC with different cut-off values
6. Acknowledgements
We thank Mr. James Robert Valera for his assistance in editing this manuscript and all the staff for their care of the patients and their contributions to this study.