ABSTRACT
As the number of COVID-19 patients has increased worldwide, many efforts have been made to find common patterns in CT images of COVID-19 patients and to confirm the relevance of these patterns against other clinical information. The aim of this paper is to propose a new method that allowed us to find patterns which observed on CTs of patients, and further we use these patterns for disease and severity diagnosis. For the experiment, we performed a retrospective cohort study of 170 confirmed patients with COVID-19 and bacterial pneumonia acquired at Yeungnam University hospital in Daegu, Korea. We extracted lesions inside the lungs from the CT images and classified whether these lesions were from COVID-19 patients or bacterial pneumonia patients by applying a deep learning model. From our experiments, we found 20 patterns that have a major effect on the classification performance of the deep learning model. Crazy-paving was extracted as a major pattern of bacterial pneumonia, while Ground-glass opacities (GGOs) in the peripheral lungs as that of COVID-19. Diffuse GGOs in the central and peripheral lungs was considered to be a key factor for severity classification. The proposed method achieved an accuracy of 91.2% for classifying COVID-19 and bacterial pneumonia with 95% reported for severity classification. Chest CT analysis with constructed lesion clusters revealed well-known COVID-19 CT manifestations comparable to manual CT analysis. Moreover, the constructed patient level histogram with/without radiomics features showed feasibility and improved accuracy for both disease and severity classification with key clinical implications.
I. INTRODUCTION
The SARS-CoV-2 pandemic virus originated in Wuhan, China in 2019, has spread rapidly to several countries [1]. RT-PCR of viral nucleic acid is regarded as the reference standard for the diagnosis of COVID-19, but chest CT examination is mostly recommended for evaluating severity and treatment efficacy given the primary involvement of the respiratory system. Also, CT imaging can be effective for early screening compared to RT-PCR that has shown low sensitivity for early detection [2]–[4]. Therefore, there is an urgent need for fast and accurate diagnostic tests other than RT-PCR.
There are well-known features of COVID-19 often observed in CT imaging such as GGOs distributed in the peripheral or posterior lungs [5]–[7]. However, these patterns are often limited and contribute to the challenge of distinguishing COVID-19 from other pneumonia types [8], [9]. Furthermore, since COVID-19 CT manifestations are often observed with mixed or subtle radiological differences, accurate description is challenging even when referencing Fleischner Society listed terms [10].
Recent advances in artificial intelligence with deep learning have shown success in the medical imaging community given the robust feature extraction capability of deep networks [11]. Herein, we analyzed chest CT scans from 73 COVID-19 and 97 bacterial pneumonia patients from Daegu, Korea. We propose a deep learning-based framework to create accurate descriptions of CT manifestations related to COVID-19. Specifically, we segmented lung and lesion regions via a segmentation model and then trained a classification model using all lesion patches and assigned them to one of two categories i.e. COVID-19 or bacterial pneumonia. The features extracted by the classification model were clustered into 20 groups via a K-means clustering algorithm [12]. Three experts confirmed that the patterns displayed in each cluster were clinically related to the typical findings of COVID-19.
For each patient, we further constructed a histogram of lesion patches given information of the 20 clusters to obtain a single representative feature vector per 3D CT image alongside radiomics features via concatenation. Finally, the features were employed to train classifiers for (i) distinguishing COVID-19 from bacterial pneumonia patients, and (ii) to assess disease severity (non-severe/severe) cases to confirm the generalization ability of the features. For the first task, the proposed method showed a 3.7% improvement over conventional majority based classification with deep learning, whereas 12.5% was reported for the severity classification task further highlighting the benefit of the proposed feature extraction strategy with significant margins observed across several metrics.
II. MATERIALS AND METHODS
A. STUDY DESIGN AND SUBJECT
We performed a retrospective cohort study of CT scans of 73 patients with COVID-19 infection obtained between February 2020 to March 2020, and 97 patients with bacterial pneumonia between March 2012 to February 2014 at Yeungnam University Medical Center, in Daegu, Korea. This study was conducted in accordance with the tenets of the Declaration of Helsinki and was reviewed and approved by the Institutional Review Board of Yeungnam University Hospital (YUH IRB 2020-05-030). The requirement for informed consent was waived due to the retrospective study design. During the study period, all consecutive adult patients (age > 18 years) with SARS-CoV-2 infection admitted to the hospital were eligible for inclusion. SARS-CoV-2 infection was confirmed by real-time RT-PCR assay of nasal and pharyngeal swab samples. Severity was defined as a composite outcome of acute respiratory distress syndrome (ARDS), intensive care unit admission, or death. ARDS was diagnosed according to the Berlin definition [13]. Figure 1 shows the flow chart of data collection, exclusion and splitting ratios applied for training and evaluation.
Flow chart.
B. IMAGING PATTERN ANALYSIS USING DEEP LEARNING
To analyze the COVID-19 manifestations using deep learning, the proposed framework consists of three key modules i.e. (a) lung and lesion segmentation, (b) deep feature extraction, and (c) K-means clustering modules, respectively. Figure 2(a) shows a diagram of the framework.
Diagram of the modeling framework. For analyzing the imaging patterns, the segmentation models were employed to extract regions of interest of the lung and lesions in the chest CT scans. Then, the ResNet50 model takes lesion cropped patches as input and returns a label prediction and a 2048-dimensional feature vector. Next, the K-means clustering algorithm was used to cluster 2D lesion features into 20 groups. To verify the usability of these patterns, (i) majority voting; (ii) constructed histogram with SVM; and (iii) Radiomics features (with/without constructed histogram) with SVM were employed. Disease and severity classification tasks were performed.
A Mask-cascade-RCNN-ResNeSt-200 with deformable convolution neural network (DCN) architecture was employed in module (a) to extract the lung and lesion regions in the chest CT scans [14], [15]. The lung segmentation model was trained and evaluated on a total 51,978 manually segmented slices (train: 50,756, test: 1,222) from two public datasets (NSCLC, 20cases) [16], [17]. For the lesion segmentation task, 6,971 manually segmented slices (train: 5,854, test: 1,117) from three publicly available datasets were used i.e. (20cases, MosMed, MSD) [17]–[19]. Then, for the data from Yeungnam University Medical Center, lesions were extracted using the trained models. The patches smaller than 13mm were not used to avoid misclassification caused by wrong segmentation or noise such as motion artifacts, as this may have a negative effect on subsequent analysis.
The deep neural network in module (b) employed a ResNet50 model trained to differentiate lesion patches of COVID-19 from those of bacterial pneumonia patients [20]. The model took a lesion cropped patch as input and returned a label prediction. A total of 12,235 lesion patches (train: 6,181, test: 6,054) from 170 patients were employed. A 2048-dimensional feature vector extracted from the intermediate layer of the ResNet50 model used in the clustering phase, i.e., module (c).
In module (c), the K-means algorithm was applied to cluster the lesion features into 20 groups. To profile the typical or relatively atypical imaging features of COVID-19, a total of 12,235 lesion patches from 170 patients were represented in two-dimensional space via a t-distributed Stochastic Neighbor Embedding based reduction of the 2048-dimensional feature vectors [21]. The lesion images in each cluster initially grouped by K-means were later manually described by one radiologist (Jongsoo Park) using imaging terms, wherein three pulmonologists (Kyung Soo Hong, Jong Geol Jang, and June Hong Ahn) evaluated the descriptions and reached a consensus.
C. VERIFICATION OF SELECTED IMAGING PATTERNS
Patient-level diagnosis can be achieved by aggregating the lesion-level predictions using majority voting. However, in this way, it is difficult to analyze what patterns the patient has and which combination of patterns are highly correlated with the disease or severity. To further quantify the diagnostic performance of the extracted imaging patterns, we constructed a histogram of lesion types using the clusters obtained via module (c) for each CT. Figure 2(b) shows a diagram of the evaluation framework. In addition, the constructed histograms were analyzed using an independent two-sample t-test to evaluate the significance of each cluster regarding: (i) COVID-19 and bacterial pneumonia patients, (ii) severe and non-severe cases. A two-tailed P <0.05 was taken to indicate statistical significance. All statistical analyses were performed using Scipy (1.5.0, https://www.scipy.org/). Furthermore, we compared the mean histograms to confirm cluster relevance using the mean of each histogram in each group i.e. COVID-19, bacterial pneumonia, severe, and nonsevere. The cluster which showed high statistical significance (P <0.001) indicated a significant value differences when performing mean histogram comparison.
Furthermore, we compared the classification accuracy between majority voting of the lesion-level predictions and SVM based inference which was trained on a combination of the proposed histogram with radiomics features. A total of 107 features were extracted using radiomics from each CT volume, including first-order statistics, shape-based features, etc. Finally, 20 features from our proposed histogram and 107 features from radiomics features were combined. We compared the performance with/without the combination of the radiomic features to verify effectiveness.
III. RESULTS
A. DEMOGRAPHIC AND CLINICAL CHARACTERISTICS
Hospitalized patients with confirmed COVID-19 (73 patients) and bacterial pneumonia (97 patients) were included in this study (Fig. 1). Baseline characteristics of all patients are summarized in Table 1. The patients in the COVID-19 group were older than the patients in the bacterial pneumonia group (58.70 ± 16.49 vs. 39.86 ± 8.69, p <0.001). 36 patients (49.3%) of COVID-19 and 55 patients (56.7%) of bacterial pneumonia were male. Body temperature (37.21 ± 0.67 vs. 37.76 ± 0.92, p <0.001), and heart rate (86.70 ± 13.94 vs. 94.37 ± 16.91, p =0.020) were significantly lower in patients with COVID-19. Systolic blood pressure (128.55 ± 19.44 vs. 115.71 ± 19.42, p <0.001), and diastolic blood pressure (80.74 ± 12.15 vs. 71.42 ± 13.52, p <0.001) were significantly higher for COVID-19 patients.
Characteristics of the study participants with COVID-19 and bacterial pneumonia.
B. DEEP CHEST CT MANIFESTATIONS
We summarize the deep chest CT manifestations observed in Table 2, with lesion patches visualized in Figure 3, respectively. Typical COVID-19 CT manifestations such as GGOs with interlobular septal thickening in the peripheral lungs were observed in clusters 4 and 17 (See Figure 4(a)). On the other hand, manifestations typical to bacterial pneumonia such as crazy-paving appearance in the posterior lungs were observed in clusters 5 and 10, as presented in Figure 4(b). These typical clusters showed P-values less than 0.001.
Summary of lesion cluster with imaging description term and P-value results. The t-test were performed for diseased and severe groups, respectively.
Visualization of the lesion patches from 170 patients into two-dimensions. (a) K-means clusters (20 groups) were used for distinction. Different color edge represents different groups. (b) True diagnosis was used for distinction. Purple color represents non-severe COVID-19, magenta represents severe COVID-19, and green for bacterial pneumonia.
(a) Typical COVID-19 CT manifestations - GGOs with interlobular septal thickening in the peripheral lungs - were observed in cluster 4 and 17. (b) bacterial pneumonia CT manifestation – crazy-paving appearance in the posterior lungs – observed in cluster 5 and 10. (c) Cluster 8 showed the lowest t-test P-value <0.001 for diseased and severe groups which is the typical pattern for severe COVID-19 patients i.e. diffuse GGOs in the central and peripheral lungs. (d) Bacterial pneumonia CT manifestation – extensive consolidation with air-bronchogram – was observed in cluster 14. These clusters showed a t-test P-value <0.001. Lesions were colored based on K-means clustering result.
Notably, 15 out of the 20 clusters showed key characteristics for the discrimination of COVID-19 from bacterial pneumonia with P-values less than 0.001. Among 15 clusters, three clusters with P-values less than 0.001 were shown to be key for severity classification. Moreover, two clusters could classify both COVID-19 and severe patients, with one of the clusters (#8) showed diffuse GGOs in the central and peripheral lungs which represents a typical severe COVID-19 CT manifestation. Figure 4(c) presents the severe COVID-19 patient’s CT manifestations.
In principle, the observed relative differences between lesion clusters are key to better understand the entire lesion distribution. As shown in Figure 4(d), extensive consolidation with air-bronchogram was observed in severe bacterial pneumonia patients. Compared with the severe COVID-19 patients who mainly showed diffuse GGOs, we could easily distinguish the difference in patterns observed in the later stages of disease onset.
To confirm cluster relevance, we compared the mean histograms of different tasks as presented in Figure 5. The clusters showed high significance (P <0.001) in Table 2 also showed a significantly different values in mean histogram comparison. Notably, the mean histogram of COVID-19 can be discriminated from bacterial pneumonia by comparing the considerably increased values in cluster 1, 4, 7, 12, and 17. On the other hand, the mean histogram of bacterial pneumonia can be discriminated from COVID-19 by comparing the significantly increased values of cluster 3, 6, 9, 11, 13, 14, and 18. In addition, the mean histogram of the severe COVID-19 patient shows consistently increased values in cluster 8 compared to the mean histogram of the non-severe COVID-19 patient. This indicates that the diagnosis is highly relevant to the presence of a lesion in a certain cluster, and the patient with a lesion in cluster 8 can be considered as a more severe COVID-19 patient than patients with lesions in other clusters. In summary, the cluster interpreted by the radiologist as being highly relevant to typical COVID-19 and bacterial pneumonia patterns showed high correlation with P-value significance in statistical analyses, with notable differences observed in mean histogram comparison.
Mean histograms i.e. (a) mean histogram of COVID-19; (b) mean histogram of bacterial pneumonia; (c) mean histogram of severe COVID-19 patients; (d) mean histogram of non-severe COVID-19 patients.
C. DISEASE AND SEVERITY CLASSIFICATION
In Table 3, we present disease classification accuracy across several metrics under different settings. First, the baseline method based on majority voting of predictions achieved 87.5% accuracy for COVID-19 patient classification. For the SVM classifier, we consider two scenarios, (i) with histogram features only or (ii) with radiomic features only. Notably, the classifier trained under the first setting achieved 88.7% accuracy; a minor improvement over the baseline, whereas in setting (ii) - 81.25% was reported, a considerable decrease from the other models. However, when the SVM classifier was trained with both radiomics and histogram features, accuracy significantly improved to 91.2% from 87.5%. This further shows that the features learned by deep learning are highly robust for accurate patient diagnosis. In addition, the constructed histograms can accurately express the correlation of lesion features observed in each patient and/or can represent some combination of patterns that could lead to a severe outcome of a disease diagnosis. In comparison to the naïve aggregation method baseline i.e. majority voting, often highly limited in patient-level representations; the proposed deep feature representation highlights several key advantages.
Test accuracy of COVID-19 diagnosis.
Table 4 shows the severity classification accuracy. In terms of accuracy alone, SVM classifiers that only use either histogram or radiomics only achieved the same performance, i.e. 82.5% for both, with key differences noted in the sensitivity and specificity of the models. On the other hand, when the features were combined considerable improvement (+12.5%) was noted i.e. 95% accuracy, surpassing previous models. This may be attributed to the fact that the histogram is not able to represent the absolute size difference of lesions, thus the combination mitigates the issue. Rather than using a single feature alone, the combination of features proved to be invaluable for diagnosis.
Test accuracy of severity COVID-19 classification.
D. LUNG AND LESION SEGMENTATION
The lung and lesion segmentation model achieved a dice coefficient score of 97.18%, and 78.06%, respectively. Figure 6 shows the lesion segmentation results around the average dice coefficient score of 78.06%, which helps to qualitatively understand the accuracy of lesion segmentation.
Representative slices which showed an average dice coefficient score (78.06%) for lesion segmentation i.e. (a) slice showed a dice coefficient score of 77.08%; (b) slice showed a dice coefficient score of 78.31%. Red color represents a manual segmentation by human and green color represents an automated segmentation by the trained segmentation model. Due to the ambiguity of lesion boundaries, an average dice score of 78.06% showed satisfactory lesion segmentation results for the deep feature extractor.
IV. DISCUSSION
Our deep chest CT analysis is in accordance with various CT findings of COVID-19 patients reported in literature. Chung et al. reported that typical CT findings of COVID-19 include bilateral pulmonary GGO and consolidative opacities which sometimes have a rounded morphology and are distributed in the peripheral lung [22]. Song et al. reported that pure GGO or GGO with reticular and/or interlobular septal thickening with predominant distribution in the posterior or peripheral lung involvements were observed in COVID-19 patients [5]. Caruso et al. highlighted the presence of the peripheral GGOs associated with multilobe and posterior lung involvement in COVID-19 patients in Italy [23]. Similar patterns were also observed in our research i.e. GGO distributed in the peripheral lungs in cluster 4 and 17, with multifocal GGOs with round morphology in cluster 1 and 7, respectively. Regarding the common findings of severe COVID-19, Pan et al. report that an increase in GGO, consolidative opacities, and interstitial septal thickening was noted, and Song et al. reported that a significantly more GGOs including pure GGO and a GGO with reticular and/or interlobular septal thickening was observed in the later stages of COVID-19 [5], [24]. An increase in GGOs in the central and peripheral lung was observed in our study in cluster 8, therefore, the published CT findings of severe COVID-19 patients show high consistency with our study.
Radiomic features are considered a useful general purpose analysis technique, i.e. for distinguishing the lung nodules (malignant versus benign) or hospital stay (severity) prediction [25], [26]. However, several limitations exist and features alone are often insufficient to distinguish between diseases when subtle radiological differences are observed in the image. Discriminating COVID-19 from bacterial pneumonia is regarded as one of the exemplars of such challenges. Here, our method shows the benefit of using deep learning to obtain more robust representations that are more clinically relevant to key imaging characteristics for COVID-19 diagnosis. We quantitatively show that the constructed histogram better captures the overall statistics of the lesion features. Moreover, the SVM classifier can diagnose diseases or patient’s severity more accurately than the radiomics features alone.
Our method has two notable advantages compared to common deep learning algorithms; interpretability and generalized representation. Common deep learning methods are limited in interpretability even though they can visualize the important regions using heatmaps [27], [28]. Our method can explain the reasons of diagnosis by checking the presence of specific patterns represented in the patient’s histogram. We verified the key patterns with mean histograms for each disease and severity group and found the important key diagnostic imaging patterns in accordance with published literature. This indicates that our method is safer and more transparent for medical assistance. Moreover, although the proposed feature learning model was not trained to classify severity among patients, the obtained features are fairly generalized for severity classification. The imaging features were divided into 20 clusters and verified by radiologists using imaging terms i.e. an independent representation of a specific diagnosis. The constructed histogram can be used for general diagnosis regardless of the trained diseases, thus the constructed histogram showed considerable diagnostic accuracy for severity classification.
The proposed framework was trained only on a single institute cohort, therefore current models may fail to accurately represent unobserved cohort CT manifestations. Moreover, the demographic characteristics of the disease group showed statistical significance. Even though these differences do not affect the CT imaging features, our method will be evaluated on multi center data in future for verification. Additionally, the K-means algorithm requires an arbitrary number of groups, it was set to 20 in this study even though it may not be the optimal value. Applying better clustering algorithms and selecting an optimal number of clusters will be the subject of future research.
V. CONCLUSION
In conclusion, the CT images of COVID-19 in Daegu, Korea were grouped into 20 clusters. These groups were analyzed and compared with the patterns described in literature. To verify the effectiveness of these clusters, we performed two classification tasks by constructing histograms from the clusters. We confirmed the correlations of the image patterns extracted by the proposed method are more relevant to the clinical setting than the common methods which use radiomics or naïve deep features.
Data Availability
N/A
APPENDIX A
Zoomed visualization of the lesion patches.
Footnotes
This work was supported by the Yeungnam University Research Fund (2020) and the National Research Foundation of Korea (NRF) grant funded by the Korean Government (MSIT) (No.2019R1C1C1008727).
REFERENCES
