Abstract
We propose a method by integrating image visibility graph and deep neural network (DL) for classifying COVID-19 patients from their chest X-ray images. The computed assortative coefficient from each image horizonal visibility graph (IHVG) is utilized as a physical parameter feature extractor to improve the accuracy of our image classifier based on Resnet34 convolutional neural network (CNN). We choose the most optimized recently used CNN deep learning model, Resnet34 for training the pre-processed chest X-ray images of COVID-19 and healthy individuals. Independently, the preprocessed X-ray images are passed through a 2D Haar wavelet filter that decomposes the image up to 3 labels and returns the approximation coefficients of the image which is used to obtain the horizontal visibility graph for each X-ray image of both healthy and COVID-19 cases. The corresponding assortative coefficients are computed for each IHVG and was subsequently used in random forest classifier whose output is integrated with Resnet34 output in a multi-layer perceptron to obtain the final improved prediction accuracy. We employed a multilayer perceptron to integrate the feature predictor from image visibility graph with Resnet34 to obtain the final image classification result for our proposed method. Our analysis employed much larger chest X-ray image dataset compared to previous used work. It is demonstrated that compared to Resnet34 alone our integrative method shows negligible false negative conditions along with improved accuracy in the classification of COVID-19 patients. Use of visibility graph in this model enhances its ability to extract various qualitative and quantitative complex network features for each image. Enables the possibility of building disease network model from COVID-19 images which is mostly unexplored. Our proposed method is found to be very effective and accurate in disease classification from images and is computationally faster as compared to the use of multimode CNN deep learning models, reported in recent research works.
Significance An integrative method is proposed combining convolutional neural networks and 2D visibility graphs through a multilayer perceptron, for effective classification of COVID-19 patients from the chest x-ray images. In our study, the computed assortative coefficient from the horizontal visibility graph of each wavelet filtered X-ray image is used as a physical feature extractor. We demonstrate that compared to Resnet34 alone, our proposed integrative approach shows significant reduction in false negative conditions and higher accuracy in the classification of COVID-19 patients. The method is computationally faster and with the use of visibility graph, it also enables one to extract complex network based qualitative and quantitative parameters for each subject for additional understandings like disease network model building and its structures etc.
1. Introduction
SARS-CoV-2 virus is highly infectious and spreads faster, affecting the respiratory organs like lungs, with the respiratory tracks developing various breath related symptoms in the patients. The severity of the disease in the humans is based on the spread of the infection to the respiratory organs. Patients suffer with heavy cough, high fever, muscle/body pain, sore throat, loss of sensation for taste and smell, headache, fatigue, and shortness of breath. In case of severe infection, the patient’s oxygen saturation level drops drastically bringing more medical complications requiring immediate oxygen support and/or intensive medical care. The disease is named as COVID-19 by the World Health Organization which declared it as a Pandemic [1-3]. Across the globe, it is observed that proper social distancing, wearing of masks covering nose and mouth, and proper sanitization effectively controls the spread. It is also consistently observed that breaking the chain of the infection by imposing lockdown restricting human movement and imposing COVID-19 containment protocol effectively controls the spread in case of an infection outbreak [4].
Continuous lockdown adversely affects the economic activities and GDP of a nation. It also severely impacts the livelihood of the population working as daily wage laborers, in unorganized sectors, self-employment businesses and for the citizens below poverty line. Hence it is essential to balance the economic activities ensuring all sections of the society are able to sustain their lives, while the nation effectively manages and controls the virus spread. More scientific approaches like early detection, test automation and other tools would help administration effectively manage and control the situation [5-6]. Various nations have also started vaccination programs to increase the human body immunity to fight against the coronavirus and its mutant strain variants. Another approach of effectively fighting against the virus is to test and diagnose the disease early so that self-isolation is maintained to further stop the virus spread and also effective medication is started early for the patient to stop the infection spread within the body [7]. Nearly 251 million people across the globe have been infected with the coronavirus with more than 5 million deaths. During surge in the virus inspection, healthcare infrastructures face acute shortage of medicines, radiology test facilities, medical oxygen, medical equipment and ICU beds to cater the surge in the high demand challenging test and hospital facility. Generally rapid antigen test (RAT), reverse transcription-polymerase chain reaction (RT-PCR) tests are performed for initial diagnosis of COVID-19. Many instances these tests showed false negatives hence are not much reliable [8-10]. Hence clinicians prefer chest radiology tests to check the lungs image for the COVID-19 Pneumonia. As computed tomography (CT) scans subject more radiation exposure for the patients with limited available facilities compared to X-ray hence X-ray is preferred for initial investigation. For a mass scale community infection, chest X-ray is a good, low-cost, first-look diagnostic tool with quick results. Also, availability of handheld X-ray devices allows easy access and early diagnosis in rural areas hence improves test penetration for effective control of infection spread through early detection.
Various machine learning and deep learning tools are used for classification of COVID-19 patients from their chest X-ray images like Random Forest, Support Vector Machine, Artificial Neural Networks (ANN), convolutional neural network (CNN) [11-20]. Generally, accuracy of various deep learning methods varies from 78-98% for classifying the COVID-19 patients based on their X-ray images [21-31]. Recently proposed MSSIM based method with the use of a simple classification learner showed 97.7 % accuracy for detecting COVID-19 X-ray images with lower false negative case [32]. As the method is comparing between two images to quantify the differences hence generates large data points due to cross-correlation analysis while using the same given dataset, making robust and accurate training for the classification learner. In the present study, we aim to use X-ray images of the patients to find characteristic features for classification using DL without use of comparison-based analysis as used in MSSIM method. Inspired by this, we propose here a new method by integrating CNN and visibility graph for faster computation and accurate classification with low false negative compared to currently used deep learning algorithms for identification of COVID-19 patients from X-ray image. Our choice of Resnet34 is due to the fact that, it is the most optimized and recently analyzed deep convolutional neural networks [33-34] that is used for image classification, object detection, image segmentation applications as it resolves the inherent problem of degrading gradient descent through residual blocks adding the skip connections in the identity mapping. The section 2 of the manuscript details the chest x-ray data and the methods used in the analysis while section 3 of the manuscript discusses the results from our analysis. Section 4 of the manuscript gives our conclusion and inference to the work.
2. Materials and Methods
The COVID-19 X-ray image database [35] is used to obtain 500 X-ray images of patients diagnosed with COVID-19. 500 healthy subject’s Chest X-rays is obtained from the open-source database [36]. These pre-processed images are used for our CNN analysis. Further for our visibility graph analysis, these X-ray images are resized to make them of equal pixel size after converting to grayscale first. The images were then resized to 1024×1024 and these image matrices are of data type double. We applied 3 label Haar wavelet decomposition and considered the approximation coefficients for our visibility graph analysis. Here to compress the image, we used 2D Haar discrete wavelet transform (HDWT). Generally, Haar wavelet transform is used to perform lossy image compression to ensure the compressed image retains its quality. This is one of the efficient procedures to perform lossless and lossy image compression as it uses averaging and differencing values in an image matrix to produce a matrix which is mostly sparse having less non-zero element in the matrix [37]. Haar wavelet transform utilizes a rectangular window for sampling. In the first label decomposition, a window width of two is used and the width doubles at each step until the window encompasses the entire data on hand. Each decomposition generates a new time series and a set of coefficients where the new time series is the average of the previous label time series over the sampling window and the coefficients represent the average change in the sampling window. Let us assume a time series {xi, xi+1, xi+2,…} then the Haar wavelet coefficient is defined as:
The Haar scaling function is written as
The Haar scaling function gives the average version of the data set and is half the size of the input time series. As the analysis procedure is recursive, the average or the smoothed data becomes the input for the next label of the wavelet transform. Unlike other wavelet functions, in Haar wavelet transform, it preserves the average in the smoothed values. The obtained average coefficient from Haar wavelet in our visibility graph analysis contains all information about the image and reduces matrix dimension to 128×128 size allowing extreme fast computation by reducing months of time required in the computation for such image visibility graph analysis to a few hours. Fig. 1 shows X-ray images of a COVID-19 subject obtained from the database along with a healthy chest image. Fig. 2 represents the sample plot of the approximation coefficient obtained from 2D Haar wavelet for COVID-19 and healthy subjects.
Further these obtained approximation coefficient matrices from Haar wavelet for each image are considered in image visibility graph analysis [38]. If the analyzing matrix is a NxN for the X-ray image I where Iij Є □ then the image visibility graph (IVG) will have N2 nodes. Now each node can be labelled by the indices of its corresponding datum Iij in a manner that two nodes ij and i’j’ are linked if (i = i’) V (j = j’) V [(i = i’ +p)□(j = j’ +p)], for some integer p, and if Iij and Ii’j’ are linked in the visibility graph defined over the ordered sequence which includes ij and i’j’. We define the visibility graph (VG) as an undirected graph of n nodes, where each node i is labelled according to the time order of its corresponding datum xi. For visibility graph, two nodes i and j where i < j are connected by an undirected link if and only if a straight line connecting xi and xj can be drawn without intersecting any intermediate datum xk for i < k < j and also the convexity criterion is fulfilled: xk < xi + ((k-i)/(j-i)) [xj - xi]; □k : i < k < j. For horizontal visibility graph (HVG) the two nodes i, j where i < j is connected by a link if and only if a horizontal line can be drawn connecting xi and xj such that it does not intersect any intermediate datum xk for i < k < j. Also, we connect i and j in the HVG if ordering criterion is fulfilled: xk < inf(xi, xj), □k : i < k < j. The image horizontal visibility graph (IHVG) obeys the same set of conditions like IVG [39-41].
We computed graphs from the coefficient matrices of the images using IHVG, where pixels are considered as nodes and the nodes are connected if they lie in a specific direction i.e. rows, columns, diagonals as per the defined HVG visibility criteria. We obtained 500 IHVG graphs for COVID-19 patients and 500 for the healthy subjects. Then from each IHVG graph the assortative co-efficient is computed as a physical feature extractor and used along with our earlier obtained Resnet34 results through a multilayer perceptron (MLP) to obtain the final classification results. As the graph for each image is available, we could obtain qualitative and quantitative characteristics like degree distribution, image descriptor patch frequency, assortative coefficient etc from the graph for characterization studies and further for possible disease network model development.
In our Resnet34 model, we normalized the pixel values of the input images using mean values [0.485, 0.456, 0.406] and the standard deviation values [0.229, 0.224, 0.225] then further the values were rescaled in range of -2 to 2. In CNN, image normalization is employed for faster convergence, easy training hence faster learning speed with stable gradient descent. Image augmentation is used in CNN model to improve the network generalization ability however we didn’t use in Resnet34 model to avoid data bias due to its use in our integrative model with image visibility graph.
We used random forest classifier for the assortative coefficients obtained from visibility graph and integrating with Resnet34 results through MLP. The classifier was chosen based on the performance from host of different classifier learners. Among various CNN models analyzed like Resnet18, Resnet50, we found Resnet34 to be performing better. We used 224×224×3 input layers with 50 epochs in our Resnet 34 model. We modified the bottom fully connected (fc) layer of our Resnet34 model with a two-neuron fc layer followed by a softmax activation output layer for computing the probabilities. The learning rate of the Resnet34 model is set to 0.001 and uses cross-entropy validation. For integrating the Resnet34 with visibility graph, we made use of an MLP classifier having relu activation function, adam optimizer with 100 hidden layers for maximum iteration of 500.
3. Results and Discussion
From the visibility graph analysis, IHVG graphs were obtained for all preprocessed image matrices. A sample IHVG graph is shown in Fig 3. To understand various qualitative and quantitative properties of these graphs, we counted the repetitions of small subgraphs in the IHVG associated to a given input image data matrix which is referred to as visibility patches (VPs). The computed patch frequency is a local image property descriptor. Visibility patch of order p is defined as, VPp which is any subgraph from the IHVG formed by a set of p2 nodes for arbitrary s,s’ satisfying the condition 1≤s, s’ ≤N-p. N equal to 128 is the size of NxN data matrix in our visibility graph analysis. In the analysis, lowest order p=3 yields nontrivial visibility patches. Visibility patches are detected when sliding a 3×3 pixel cell of stride 1 in the image extracting the corresponding IHVGs within the cell. It enables reduced checking of different combination of visibility graph motifs presence or absence hence mathematically enables tractability of visibility patches for computation. Fig 4 shows the visibility patches for 3 randomly selected X-ray images of both healthy and COVID-19 subjects.
We analyzed the degree distribution P(k) where P(k) ∼ k-λ to understand statistical properties of the networks where k is the degree of a node and is defined as the number of edges connected to a node. If the degree distribution of the network, follow a power law distribution then we call it a scale-free network which is characterized by the nodes of the networks that are linked to a significant fraction of the total number of edges of the network. Log-log plot of the degree distributions P(k) for a sample IHVG is shown in Fig 5. We observe that for both healthy and COVID-19 patients the obtained network is scale free.
We computed assortative coefficients for each obtained IHVG which is used along with Resnet34. A complex network is defined as assortative mixing if higher degree nodes of the network tend to be connected to other higher degree nodes whereas in disassortative mixing the high degree nodes attach to low degree nodes only. Computationally, these assortative complex networks remove its highest degree nodes efficiently compared to the disassortative networks. For an undirected network, the assortative coefficient is computed as: where ji, ki are the degrees of the two nodes at the ends of the ith edge where i=1,2, upto M. Assortative coefficient values vary between -1 to 1 where assortative networks have positive r values while disassortative networks have negative r values. For both COVID-19 and healthy subjects in our analysis, the obtained assortative coefficient values are always positive indicating that all the networks for disease and healthy are assortative.
The obtained 1000 assortative coefficients for both COVID-19 and healthy are labeled with its attribute category for random forest classifier and are further used along with Resnet34 in a MLP classifier as a quantitative physical parameter feature extractor for improving classification performance of standalone Resnet34 as visualized from the confusion matrix. Fig 6 illustrates our system study approach.
It is worth emphasizing that when the sample size of chest X-ray images increases as in our case to 1000 images, the resnet34 model without image augmentation gives 96% classification accuracy unlike earlier reported accuracy of 98.33% with 406 images [33]. We avoided image augmentation in our CNN model to overcome the data bias error that might occur in testing. As shown in Fig6, we make use of Resnet34 on the X-ray images to classify the COVID-19 and healthy subjects. In a parallel approach, from the preprocessed images passing through Haar wavelet visibility graph was introduced to learn the structural connectivity from the obtained network and then random forest classifier is used on VG assortative coefficient to classify the images. Then we integrate both Resnet34 and VG using a MLP considering the predicted labels from both for the final classification result of MLP. In our model, we used 1000 chest X-ray images of which 500 were for COVID-19 while 500 of healthy. We used 800 of these images equally from both the class for training the Resnet34 model and the random forest classifier using assortative co-efficient from visibility graph. The remaining 200 is used for validation. The predicted labels from random forest and Resnet34 were fed to the MLP classifier in which 100 data points were used for training while 100 were used for testing to obtain the final accuracy of our model. Table 1 summaries performance of our proposed multi-mode ensemble model with recently developed models. We observe our proposed multi-mode ensemble with introduction of VG as a unique characteristic feature that increases the accuracy by 2% compared to use of Resnet34 model alone. Similarly, the F1-score, sensitivity is increased by almost 1.8% compared to the Resnet34 model alone. Importantly, from the confusion matrix as shown in Table 2 we also observe our integrative model decreases false negative cases proportionately. The Fig 7 shows our proposed method training and test accuracy and loss. Our integrative model loss against epoch is very close to 0 while it is 0.4 for Resnet34. The Fig 8 shows the receiver operating characteristic (RoC) curve for our newly proposed model.
4. Conclusion
In conclusion, we have shown our integrative multi-model ensemble method combining Resnet34 CNN model and 2D visibility graph helps in better classification of COVID-19 chest X-ray images compared to a CNN model alone or in combinations as reported in prior research work. Also, our method is computationally very fast and with introduction of Haar wavelet it drastically reduced computation time for image visibility graph and its associated parameter calculation while improving classification performance. With introduction of visibility graph various complex network qualitative and quantitative parameters for the subject image could be obtained and a disease network model could be built for future research on COVID-19.
Data Availability
The COVID-19 X-ray image database [35] is used to obtain 500 X-ray images of patients diagnosed with COVID-19. 500 healthy subjects Chest X-rays is obtained from the open-source database [36].
Authors’ contributions
MPal conceived the idea and conceptualized it, developed the model concept and its code, performed the data pre-processing and image processing analysis, wrote and reviewed the manuscript. YT performed the CNN and MLP related code enhancement, analysis, model optimization and contributed to the manuscript writing. TVR developed primary model framework and code. PSRA debugged the code and performed CNN analysis. PKP conceptualized the idea of using haar wavelet with visibility graph, mentored the work and reviewed the manuscript.
Acknowledgements
This research work did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. The author alone is responsible for the content and writing of the paper.