AI/ML Models to Aid in the Diagnosis of COVID-19 Illness from Forced Cough Vocalizations: Results and Challenges of a Systematic Review of the Relevant Literature ================================================================================================================================================================== * K. Kelley * A.A. Sakara * M. Kelley * S. C. Kelley * P. McLenaghan * R. Aldir * M. Cox * N. Donaldson * A. Stogsdill * S. Kotchou * G. Sula * M.A. Ramirez ## Abstract From a comprehensive and systematic search of the relevant literature on signal data signature (SDS)-based artificial intelligence/machine learning (AI/ML) systems designed to aid in the diagnosis of COVID-19 illness, we aimed to reproduce the reported systems and to derive a performance goal for comparison to our own medical device with the same intended use. These objectives were in line with a pathway to regulatory approval of such devices, as well as to acceptance of this unfamiliar technology by disaster/pandemic decision makers and clinicians. To our surprise, none of the peer-reviewed articles or pre-print server records contained details sufficient to meet the planned objectives. Information amassed from the full review of more than 60 publications, however, did underscore discrete impediments to bringing AI/ML diagnostic solutions to the bedside during a pandemic. These challenges then were explored by the authors via a gap analysis and specific remedies were proposed for bringing AI/ML technologies in closer alignment with the needs of a Total Product Life Cycle (TPLC) regulatory approach. ## 1. Introduction The onset of the SARS-CoV-2 outbreak in Wuhan, China in December 2019 and subsequent WHO declaration of a global pandemic in March 2020 has rallied workforces of every skillset to the cause. Frontline healthcare teams, supply chain employees, educational staff, and utilities personnel were called upon as Essential Workers. Anticipating the needs for processing large amounts of pandemic-related data, the Artificial Intelligence/Machine Learning (AI/ML) and Data Science (DS) communities also joined forces to contribute their talents to the response effort. AI/ML technologies were applied to the development and maintenance of numerous types of signal data signature (SDS) libraries, registries, and clinical datasets from forced cough vocalizations (FCV). Voice, breath, and FCV research already had established that neural networks coupled with feature extraction analysis by AI/ML engines could be used to identify a variety of respiratory and neurological diseases. Drawing from this prior body of work, teams across the globe began working independently to establish SDS-based software systems to aid in the diagnosis of COVID-19 illness. But how many of the resulting software-as-medical-devices (SaMDs) would have the safety and performance profiles to support a viable regulatory pathway to market? And would the application and usability of this unfamiliar technology gain acceptance for clinical use during a disaster response? To answer these questions, we prospectively planned a PRISMA 2020 systematic review of the relevant literature. Our primary objectives were to demonstrate reproducibility of the published models by building each COVID-19 diagnostic software system for which sufficient details were reported and to conduct head-to-head evaluations both across the completed models and also in comparison to our own AI/ML system. As a secondary objective, our intent was to determine a literature-derived performance goal (PG) for the completed models for comparison to our device with the same intended use. ## 2. Background Implementation of AI/ML solutions to clinical needs hinges upon strong adherence to scientific principles, good clinical practice, and the medical device regulatory process. Specific to times of public health emergency, numerous governments and regulatory agencies have authority to bring devices, drugs or biologics with a risk-to-benefit profile deemed acceptable into use for affected populations. The US-FDA Emergency Use Authorization (EUA), the WHO Emergency Use Letter (EUL), and a global array of expedited governmental and healthcare agency review pathways are all examples of routes to emergency clinical use. And key to achieving rapid acceptance of and fostering trust in newer technologies during disasters or emergencies is keeping the regulatory process central to all aspects of design, development, and deployment. In other words, documentation spanning the Total Product Life Cycle (TPLC) from inspiration to post-market surveillance. In response to the immense need for COVID-19 diagnostic testing during the pandemic, scientific teams of diverse disciplines have developed hundreds of devices, most of which were based upon established methods of laboratory analysis. There were other approaches, though, proposed by AI/ML and Data Science professionals in search of a scalable diagnostic device that did not require wet specimens or laboratory processing. Academic institutions from around the world have cataloged, analyzed, and published peer-reviewed articles on SDS libraries that were amassed both before and after December 2019. This collective wealth of data contains FCV recordings from years prior to the pandemic (not COVID-19), post-December 2019 recordings of COVID-19 negative persons as confirmed by RT-PCR (not COVID-19), and FCV recordings from persons diagnosed COVID-19 positive and verified by RT-PCR (yes COVID-19). While some author groups employed in-house collections of recordings, there are a number of publicly available SDS libraries that are identifiable from literature searches and downloadable for use in the development of SDS-based COVID-19 diagnostic software systems. ## 3. Materials and Methods Certain terms will be adopted uniformly throughout this review, to provide clarity. The term “article” will be reserved for peer-reviewed references and those obtained from pre-print servers (non-peer-reviewed) will be termed “records”. While many of the references refer to an SDS collection from FCV as a “registry”, the reviewers will use the term “library”, so as not to suggest that the same degree and quality of clinical data typically included in medical device registry exists in each SDS collection. Lastly, the shorter phrase “SDS Library” will be representative of each SDS library from FCV. ### 3.1. Literature Searches On 12 October 2021 and updated on 7 November 2021, we conducted systematic searches of the relevant literature for the purpose of presenting a comparative evaluation of AI/ML systems designed to aid in the diagnosis of COVID-19 from FCV. Searches of the peer-reviewed literature were prioritized but, given the collaborative “shareware” culture of the AI/ML and Data Science communities, pre-print servers were searched for possible contributions. EndNote 2020 was the designated reference manager and PubMed was searched via this software. Serial searches of “Any Field” in PubMed, “Full Text and Metadata” in the IEEE Xplore digital library of the Institute of Electrical and Electronics Engineers,([ieeexplore.ieee.org](http://ieeexplore.ieee.org)), “All Fields” in the arXiv open-access archive ([arxiv.org](http://arxiv.org)), and “Full Text or Abstract or Title” in bioRxiv and medRxiv ([medrxiv.org](http://medrxiv.org)) were performed using the identical search terms as listed below: * covid and classifier * covid and neural network * covid and cough and artificial intelligence * covid and cough and AI * covid and cough and machine learning * covid and cough and ML * covid and cough and classifier * cough and neural network * forced cough vocalization The results from these serial searches were combined and systematically filtered to achieve a final article pool from which all references would be evaluated for contribution to the stated objectives. Following a basic PRISMA 2020 workflow, our search methodology is illustrated below in Figure 1: ![Figure 1](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/11/14/2021.11.12.21266271/F1.medium.gif) [Figure 1](http://medrxiv.org/content/early/2021/11/14/2021.11.12.21266271/F1) Figure 1 PRISMA 2020 Flow Diagram for New Systematic Reviews (inclusive of Databases, SDS Libraries, and Other Sources) ### 3.2. Classification System for SDS Libraries Since the introduction of levels of evidence into the scientific literature, many professional organizations and journals have adopted some variation of this classification system. Diverse specialties are often asking different questions, though, and it has been recognized that the type and level of evidence needs to be modified accordingly [1]. Evidence-Based Medicine classification systems specifically are based upon research design questions that take into account prospective vs retrospective data collection, data collection methods, data sources, data verification, control design, study population sampling, diagnosis, and decision analysis. In order for comparisons to be made across the many SDS libraries used in the above references, an appropriate classification system was required. Grades of Data are another classification system, derived from the Level of Evidence (data collection design) and modified in concert with the same research design questions, but only as they apply to each data subset. Therefore, an SDS library with a Level 2b design is expected to yield Moderate Grade data. If some retrospective data were to be included in a given library, the inclusion of this data subset would reduce the grade level for the overall collection by one grade level to Low Grade data. Likewise, if some or all of the retrospective data also had incomplete clinical data, then the grade of data would be reduced by two levels (one each for retrospective and incomplete clinical data), degrading the library data to Insufficient [2, 3]. Conversely, if an SDS library with a Level 2b design yielding Moderate Grade data were to have a subset verified by two RT-PCR tests within 24 hours of the FCV recording, that subset of data would be classified as High Grade Data and the entire library would be elevated by one data grade. To permit assignment of design levels and stratification of data as reported, the Levels of Evidence design and associated Grades of Data criteria were modified according to the needs of AI/ML training, validation, and testing sets. This yielded a classification system appropriate to the topic of our review and is shown below as Table 1. View this table: [Table 1](http://medrxiv.org/content/early/2021/11/14/2021.11.12.21266271/T1) Table 1 Levels of Evidence and Grade of Data Classification System for SDS Libraries ## 4. Results From the final pool of references, all were read and evaluated across a multidisciplinary team of Data Scientists, Biotechnical Engineers, Healthcare Clinicians, Product Development and Clinical/Regulatory Affairs professionals. There were 57 references proposing a specific COVID-19 diagnostic AI/ML model or application (33 peer-reviewed articles and 24 pre-print server records) and the remaining 7 articles were Editorials and Technical notes reserved for potential contribution to the Discussion. ### 4.1. Primary Objectives Although each of the 57 references purported to include a fully-detailed AI/ML solution, only 14 contained enough information for us to attempt building the stated solution. Unfortunately, no single reference included sufficient details to build a complete model or system, rendering comparisons across models (including our own) unattainable. The most common omissions pertained to the model’s architecture or the flow of data through the model’s layers. Thus, the primary objectives of this review could not be met. ### 4.2. Secondary Objective Determination of a literature-derived PG was the secondary objective of this review and is a familiar endpoint in regulatory pathways. To realize this goal, the intent was to mine the data as reported and as trained for the individual models and calculate the PG. AI/ML software solutions meeting the 5 prospective criteria as below, defining a comparable population, were to be included in the calculation: * Commonly defined sensitivity and specificity of the model when tested with the respective test set * False positive and false negative were reported in some fashion * Test sets were of statistically significant size, based upon Exact Binomial Test * Data sets represented good quality data, defined as a Design Level 1 or 2 and Data Grade of High or Moderate * Results for PPA and NPA (directly reported or calculable from the details presented in the reference) were compared to RT-PCR for COVID-19 test results, in keeping with submission criteria for US-FDA EUA or WHO EUL There were 6 of 57 references found to have a Level of Evidence design of 2 or better, High or Moderate Grade data, and a statistically significant SDS library size provided the testing results to meet criteria (see Table 2). And of these same 6 references, only 2 were peer-reviewed articles. None of the reference met all criteria, rendering the secondary objective unattainable. Additionally, no more than 2 of the 6 references shared a common endpoint, disabling even that aspect of potential comparison. View this table: [Table 2](http://medrxiv.org/content/early/2021/11/14/2021.11.12.21266271/T2) Table 2 Articles Meeting 3 of 5 Criteria for Establishing a Literature-Derived Performance Goal ### 4.3. Gap Analysis We previously had developed a categorization system specific to our objectives and determined the level of evidence and grade of data for each of the 57 references. To emphasize aspects of each library population rather than individual articles, this time we reorganized the information as seen below in Table 3: View this table: [Table 3](http://medrxiv.org/content/early/2021/11/14/2021.11.12.21266271/T3) Table 3 SDS Libraries Represented in Current Literature Search Article Pool Looking at the review data from this vantage point, we observed that all of the SDS libraries collected by author groups themselves for the purpose of their model development (identified as In-House Collections) had Level 2a design that yielded Moderate Grade data. And only the projects employing the Coswara library (whether or not the reference was peer-reviewed) and the Israeli COVID-19 dataset ranked equally. Since the intended objectives could not be met, we reexamined the data from our review for alternative utility. The logical next steps from a regulatory approach were to complete a formal gap analysis, to focus on the issues that rendered our aims unattainable. Perhaps the challenges central to researching diagnostic AI/ML solutions for COVID-19 were representational of the greater challenges faced by AI/ML development teams entering the medical device market, particularly during a public health emergency? Answering this question and itemizing the contributing factors became our new objectives. The results of our gap analysis are summarized below in Table 4, itemizing the recurring themes identified in the body of literature assembled during this review. Each gap directly impacted our ability to meet primary and secondary objectives but also would affect the market-readiness of any medical device. Plans proposed here to mitigate or resolve these gaps were intended to apply not only to future AI/ML development publications but also to support realization of a regulatory pathway, keeping the needs of the TPLC in mind and clinical relevance at the core. View this table: [Table 4](http://medrxiv.org/content/early/2021/11/14/2021.11.12.21266271/T4) Table 4 Gap Analysis Summary View this table: [Table 4:](http://medrxiv.org/content/early/2021/11/14/2021.11.12.21266271/T5) Table 4: Summary of Andreu-Perez et al. [4] View this table: [Table 5:](http://medrxiv.org/content/early/2021/11/14/2021.11.12.21266271/T6) Table 5: Summary of Verde et al. [5] View this table: [Table 6:](http://medrxiv.org/content/early/2021/11/14/2021.11.12.21266271/T7) Table 6: Summary of Our System in Development (Pre-Trial) ## 5. Discussion ### 5.1. Employing Remedies to Bridge Gaps Data quality issues and the ultimate impact on model function underscored the most common challenges noted in both the research and editorial articles contributing to this review [60], [61]. To enhance the data quality and optimize the Grade of data, several authors suggested methods to verify symptomology and augment clinical diagnosis of subjects with suspected COVID-19 illness. Specific technologies cited for improving library and data collection design include Internet-Of-Things (IOT) sensors, mobile digital health products, and mHealth wearable devices [62]. [60] [63], [61], [64], [65], [66]. ### 5.2. Review of Level 2a, Moderate Grade, Statistically Significant Articles At first glance, the reference pool identified by our searches and reviews of the literature appeared to yield relatively complete descriptions of working COVID-19 diagnostic AI/ML models available for download or as a live working model via the Internet. Re-creation of these models began as a promising endeavor, despite the need for training, validation, and testing specifics. Each was built by our teams according to the plans presented in the respective reference but, regrettably, none of the models were reproducible as a fully deployed or fully deployable device. The recurring theme throughout our review process was that most references, at least as documented, provided more exploration than execution of diagnostic SaMD models for COVID-19. Some notable exceptions were the technological contributions offered in articles by Imran et al. [31] and Orlandic et al [28] and also the pre-print paper by Chowdhury et al. [41]. But without inclusion of a clear regulatory pathway, the authors did not bring the TPLC full-circle for the models they presented [41, 51] [28, 31]. Two additional publications of distinction were Andreu-Perez et al. [4] and Verde et al. [5]. As the only articles meeting a majority of criteria for establishing a performance goal for the AI/ML systems of interest, their respective work is summarized below and accompanied by a table of model-specific strengths and challenges. Andreu-Perez, et al [4] reported on an AI/ML system to screen for COVID-19 based upon FCV. The SDS library employed was an in-house library collected in Spain, which was categorized in the current review as Level 2a design and a statistically significant amount of Moderate Grade data. Each SDS record was analyzed by the ML and SDS processing system, filtered by a cough detector, and results are given to the user. Incoming recordings were filtered, cleaned, and then passed through their cough detector based on empirical mode decomposition (EMD). Cough information then was transformed into tensor form using Mel-Frequency Cepstral Coefficients (MFCCs), Mel-scaled spectrograms, and Linear Predictive Coding Spectrum (LPCS) coefficients. Extracted from each SDS were 33 features which were mapped onto a 3D tensor, using a CNN with three main layers stacked four times. The authors achieved a true positive accuracy of 97.18% and a true negative rate of 96.64%. However, since false positive and false negative data were not reported, Positive Percent Agreement (PPA) and Negative Percent Agreement (NPA) could not be determined. In the second of two AI/ML systems for in-depth discussion, Verde et al. [5] also employed Coswara as their source of SDS data. The authors analyzed a variety of ML algorithms to detect COVID-19 through the phonetic vowel sounds of /a/, /e/, and /o/ that are inherent to FCV as the feature for analysis. In their listing of the model performances, the Support Vector Machine (SVM) performed the best overall: achieved percentages for Accuracy, F1-Score, Specificity, Precision, and Recall were 97.07, 82.35, 97.37, 73.68, 93.33 respectively and Area Under the Curve = 0.954. But as with Andreu-Perez et al. [4], since false positive and false negative data were not reported, PPA and NPA were not calculable. ### 5.3. Introduction of Our System in Development Our AI/ML system for the diagnosis of COVID-19 illness, in common with the Level 2a is based upon the SDS from FCV. Having determined the Level 2a design and Moderate Grade SDS libraries from a systematic review of the literature, we selected Coswara as the source for statistically significant sized datasets. Incoming recordings are filtered, cleaned, and then passed through a cough detector. Once the SDS has been declared an FCV through audio analysis, it is then segmented into individual FCV segments. This is completed using hidden Markov modeling, which detects the onset and completion of each FCV within a file, standardizing the FCV for the classification models. FCV information is then transformed into tensor form using Mel-Frequency Cepstral Coefficients (MFCCs), Mel-scaled spectrograms, and Fast Formant Transformation (FFT) to yield SDS for feature analysis. Each SDS feature is analyzed separately, extracted, and mapped using an ensemble based upon successful neural network modeling techniques of 2D-CNN and RNN in the peer-reviewed literature [4] [31] [28]. Our process also employs ML algorithms to detect COVID-19 illness via the phonetic vowel sounds inherent to FCV as the feature for analysis in a manner similar to Verde et al. [5]. In anticipation of our upcoming clinical trial, we have achieved a Positive Percent Agreement (PPA) greater than 0.90 and Negative Percent Agreement (NPA) greater than 0.85 in benchtop testing. ## 6. Conclusion While unable to meet the intended objectives of this systematic review of the AI/ML system literature on SDS from FCV aiding in the the diagnosis of COVID-19 illness, the authors did complete a gap analysis that identified the principal issues and significant challenges facing this growing field of study. The consequences of such gaps directly affected not only the quality of data available in the relevant literature but also prevented analyses to assess regulatory readiness of the devices and applications presented. Acceptance of unfamiliar technology by disaster or pandemic decision makers and clinicians would require bridging of these gaps through adherence to a clear and well-documented TPLC approach to a viable regulatory pathway. ## Data Availability All data produces is available upon request from the authors ## Footnotes * Dedicated to Our Respected Fellow AI/ML COVID-19 Signal Data Signature Researchers and Healthcare Decision Makers who have died of COVID-19: M. Maleki and M.E. Thornton. * Received November 12, 2021. * Revision received November 12, 2021. * Accepted November 14, 2021. * © 2021, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NoDerivs 4.0 International), CC BY-ND 4.0, as described at [http://creativecommons.org/licenses/by-nd/4.0/](http://creativecommons.org/licenses/by-nd/4.0/) ## 7. References 1. [1].Burns, P.B., R.J. Rohrich, and K.C. Chung, The Levels of Evidence and Their Role in Evidence-Based Medicine. Plastic and reconstructive surgery, 2011. 128(1): p. 305–310. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/PRS.0b013e318219c171&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21701348&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F14%2F2021.11.12.21266271.atom) 2. [2].Granholm, A., W. Alhazzani, and M.H. Møller, Use of the GRADE approach in systematic reviews and guidelines. British journal of anaesthesia, 2019. 123(5): p. 554–559. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F14%2F2021.11.12.21266271.atom) 3. [3].Schünemann, H.J., et al., Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. Bmj, 2008. 336(7653): p. 1106–1110. [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMzYvNzY1My8xMTA2IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjEvMTEvMTQvMjAyMS4xMS4xMi4yMTI2NjI3MS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 4. [4].Andreu-Perez, J., et al., A generic deep learning based cough analysis system from clinically validated samples for point-of-need covid-19 test and severity levels. IEEE Transactions on Services Computing, 2021. 5. [5].Verde, L., et al., Exploring the Use of Artificial Intelligence Techniques to Detect the Presence of Coronavirus Covid-19 Through Speech and Voice Analysis. IEEE Access, 2021. 9: p. 65750–65757. 6. [6].Alkhodari, M. and A. Khandoker, Detection of COVID-19 in smartphone-based breathing recordings using CNN-BiLSTM: a pre-screening deep learning tool. medRxiv, 2021. 7. [7].Chang, J., S. Cui, and M. Feng, DiCOVA-Net: Diagnosing COVID-19 using Acoustics based on Deep Residual Network for the DiCOVA Challenge 2021. arXiv preprint arxiv:2107.06126, 2021. 8. [8].Chetupalli, S.R., et al., Multi-modal Point-of-Care Diagnostics for COVID-19 Based On Acoustics and Symptoms. arXiv preprint arxiv:2106.00639, 2021. 9. [9].Muguli, A., et al., DiCOVA Challenge: Dataset, task, and baseline system for COVID-19 diagnosis using acoustics. arXiv preprint arxiv:2103.09148, 2021. 10. [10].Jayachitra, V.P., et al., A cognitive IoT-based framework for effective diagnosis of COVID-19 using multimodal data. Biomed Signal Process Control, 2021. 70: p. 102960. 11. [11].Pahar, M., et al., COVID-19 cough classification using machine learning and global smartphone recordings. Comput Biol Med, 2021. 135: p. 104572. 12. [12].Tena, A., F. Claria, and F. Solsona, Automated detection of COVID-19 cough. Biomed Signal Process Control, 2022. 71: p. 103175. 13. [13].Amoh, J. and K. Odame, Deep Neural Networks for Identifying Cough Sounds. IEEE Trans Biomed Circuits Syst, 2016. 10(5): p. 1003–1011. 14. [14].Bartl-Pokorny, K.D., et al., The voice of COVID-19: Acoustic correlates of infection in sustained vowels. The Journal of the Acoustical Society of America, 2021. 149(6): p. 4377–4383. 15. [15].Hoyos-Barcelo, C., et al., Efficient k-NN Implementation for Real-Time Detection of Cough Events in Smartphones. IEEE J Biomed Health Inform, 2018. 22(5): p. 1662–1671. 16. [16].Knocikova, J., et al., Wavelet analysis of voluntary cough sound in patients with respiratory diseases. J Physiol Pharmacol, 2008. 59(Suppl 6): p. 331–40. 17. [17].Liu, J.-M., et al. Cough detection using deep neural networks. in 2014 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). 2014. IEEE. 18. [18].Monge-Alvarez, J., et al., Robust Detection of Audio-Cough Events Using Local Hu Moments. IEEE J Biomed Health Inform, 2019. 23(1): p. 184–196. 19. [19].Monge-Alvarez, J., et al., Effect of importance sampling on robust segmentation of audio-cough events in noisy environments. Annu Int Conf IEEE Eng Med Biol Soc, 2016. 2016: p. 3740–3744. 20. [20].Monge-Álvarez, J., et al., A machine hearing system for robust cough detection based on a high-level representation of band-specific audio features. IEEE Transactions on Biomedical Engineering, 2018. 66(8): p. 2319–2330. 21. [21].Pal, A. and M. Sankarasubbu. Pay attention to the cough: Early diagnosis of COVID-19 using interpretable symptoms embeddings with cough sound signal processing. in Proceedings of the 36th Annual ACM Symposium on Applied Computing. 2021. 22. [22].Pinkas, G., et al., SARS-CoV-2 detection from voice. IEEE Open Journal of Engineering in Medicine and Biology, 2020. 1: p. 268–274. 23. [23].Belkacem, A.N., et al., End-to-End AI-Based Point-of-Care Diagnosis System for Classifying Respiratory Illnesses and Early Detection of COVID-19: A Theoretical Framework. Front Med (Lausanne), 2021. 8: p. 585578. 24. [24].Coppock, H., et al., End-to-end convolutional neural network enables COVID-19 detection from breath and cough audio: a pilot study. BMJ Innov, 2021. 7(2): p. 356–362. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYm1qaW5ub3YiO3M6NToicmVzaWQiO3M6NzoiNy8yLzM1NiI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIxLzExLzE0LzIwMjEuMTEuMTIuMjEyNjYyNzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 25. [25].Lella, K.K. and A. Pja, Automatic COVID-19 disease diagnosis using 1D convolutional neural network and augmentation with human respiratory sound based on parameters: cough, breath, and voice. AIMS Public Health, 2021. 8(2): p. 240–264. 26. [26].Shimon, C., et al., Artificial intelligence enabled preliminary diagnosis for COVID-19 from voice cues and questionnaires. J Acoust Soc Am, 2021. 149(2): p. 1120. 27. [27].Mouawad, P., T. Dubnov, and S. Dubnov, Robust Detection of COVID-19 in Cough Sounds. SN Computer Science, 2021. 2(1): p. 1–13. 28. [28].Orlandic, L., T. Teijeiro, and D. Atienza, The COUGHVID crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms. Sci Data, 2021. 8(1): p. 156. 29. [29].Bhateja, V., A. Taquee, and D.K. Sharma. Pre-processing and classification of cough sounds in noisy environment using SVM. in 2019 4th International Conference on Information Systems and Computer Networks (ISCON). 2019. IEEE. 30. [30].Laguarta, J., F. Hueto, and B. Subirana, COVID-19 artificial intelligence diagnosis using only cough recordings. IEEE Open Journal of Engineering in Medicine and Biology, 2020. 1: p. 275–281. 31. [31].Imran, A., et al., AI4COVID-19: AI enabled preliminary diagnosis for COVID-19 from cough samples via an app. Inform Med Unlocked, 2020. 20: p. 100378. 32. [32].Cohen-McFarlane, M., R. Goubran, and F. Knoefel. Comparison of silence removal methods for the identification of audio cough events. in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2019. IEEE. 33. [33].Cohen-McFarlane, M., R. Goubran, and F. Knoefel, Novel coronavirus cough database: NoCoCoDa. IEEE Access, 2020. 8: p. 154087–154094. 34. [34].Miranda, I.D., A.H. Diacon, and T.R. Niesler. A comparative study of features for acoustic cough detection using deep architectures. in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2019. IEEE. 35. [35].Mydukuri, R.V., et al., Deming least square regressed feature selection and Gaussian neuro-fuzzy multi-layered data classifier for early COVID prediction. Expert Syst, 2021: p. e12694. 36. [36].Mohammed, E.A., et al., An ensemble learning approach to digital corona virus preliminary screening from cough sounds. Sci Rep, 2021. 11(1): p. 15404. 37. [37].Feng, K., et al. Deep-learning Based Approach to Identify Covid-19. in SoutheastCon 2021. 2021. IEEE. 38. [38].Melek, M., Diagnosis of COVID-19 and non-COVID-19 patients by classifying only a single cough sound. Neural Comput Appl, 2021: p. 1–12. 39. [39].Melek Manshouri, N., Identifying COVID-19 by using spectral analysis of cough recordings: a distinctive classification study. Cognitive Neurodynamics, 2021. 40. [40].Quatieri, T.F., T. Talkar, and J.S. Palmer, A framework for biomarkers of COVID-19 based on coordination of speech-production subsystems. IEEE Open Journal of Engineering in Medicine and Biology, 2020. 1: p. 203–206. 41. [41].Chowdhury, M.E., et al., QUCoughScope: An Artificially Intelligent Mobile Application to Detect Asymptomatic COVID-19 Patients using Cough and Breathing Sounds. arXiv preprint arxiv:2103.12063, 2021. 42. [42].Pahar, M., et al., COVID-19 Detection in Cough, Breath and Speech using Deep Transfer Learning and Bottleneck Features. arXiv preprint arxiv:2104.02477, 2021. 43. [43].Sharma, M., et al., Impact of data-splits on generalization: Identifying COVID-19 from cough and context. arXiv preprint arxiv:2106.03851, 2021. 44. [44].Sharma, N., et al., Coswara--A Database of Breathing, Cough, and Voice Sounds for COVID-19 Diagnosis. arXiv preprint arxiv:2005.10548, 2020. 45. [45].Xue, H., & Salim, F. D., Exploring Self-Supervised Representation Ensembles for COVID-19 Cough Classification. arXiv preprint arxiv:2105.07566., 2021. 46. [46].Akman, A., et al., Evaluating the COVID-19 Identification ResNet (CIdeR) on the INTERSPEECH COVID-19 from Audio Challenges. arXiv preprint arxiv:2107.14549, 2021. 47. [47].Bagad, P., et al., Cough against COVID: Evidence of COVID-19 signature in cough sounds. arXiv preprint arxiv:2009.08790, 2020. 48. [48].Brown, C., et al., Exploring automatic diagnosis of COVID-19 from crowdsourced respiratory sound data. arXiv preprint arxiv:2006.05919, 2020. 49. [49].Shams, A.B., et al., Telehealthcare and Covid-19: A Noninvasive & Low Cost Invasive, Scalable and Multimodal Real-Time Smartphone Application for Early Diagnosis of SARS-CoV-2 Infection. arXiv preprint arxiv:2109.07846, 2021. 50. [50].Nguyen, L.H., et al., Fruit-CoV: An Efficient Vision-based Framework for Speedy Detection and Diagnosis of SARS-CoV-2 Infections Through Recorded Cough Sounds. arXiv preprint arxiv:2109.03219, 2021. 51. [51].Chowdhury, N.K., M.A. Kabir, and M. Rahman, An Ensemble-based Multi-Criteria Decision Making Method for COVID-19 Cough Classification. arXiv preprint arxiv:2110.00508, 2021. 52. [52].Schuller, B.W., et al., The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 cough, COVID-19 speech, escalation & primates. arXiv preprint arxiv:2102.13468, 2021. 53. [53].Schuller, B.W., H. Coppock, and A. Gaskell, Detecting COVID-19 from breathing and coughing sounds using deep neural networks. arXiv preprint arxiv:2012.14553, 2020. 54. [54].Bales, C., et al. Can machine learning be used to recognize and diagnose coughs? In 2020 International Conference on e-Health and Bioengineering (EHB). 2020. IEEE. 55. [55].Chen, X., M. Hu, and G. Zhai, Cough Detection Using Selected Informative Features from Audio Signals. arXiv preprint arxiv:2108.03538, 2021. 56. [56].Deshpande, G. and B.W. Schuller, Audio, Speech, Language, & Signal Processing for COVID-19: A Comprehensive Overview. arXiv preprint arxiv:2011.14445, 2020. 57. [57].Pizzo, D.T., S. Esteban, and M. Scetta, IATos: AI-powered pre-screening tool for COVID-19 from cough audio samples. arXiv preprint arxiv:2104.13247, 2021. 58. [58].Hossain, M.Z., M.B. Uddin, and K.A. Ahmed, CovidEnvelope: A Fast Automated Approach to Diagnose COVID-19 from Cough Signals. medRxiv, 2021. 59. [59].Ritwik, K.V.S., S.B. Kalluri, and D. Vijayasenan, COVID-19 patient detection from telephone quality speech data. arXiv preprint arxiv:2011.04299, 2020. 60. [60].Coppock, H., et al., COVID-19 detection from audio: seven grains of salt. Lancet Digit Health, 2021. 3(9): p. e537–e538. 61. [61].Khanzada, A., et al., Challenges and Opportunities in Deploying COVID-19 Cough AI Systems. J Voice, 2021. 62. [62].Adans-Dester, C.P., et al., Can mHealth technology help mitigate the effects of the COVID-19 pandemic? IEEE Open Journal of Engineering in Medicine and Biology, 2020. 1: p. 243–248. 63. [63].Faezipour, M. and A. Abuzneid, Smartphone-based self-testing of COVID-19 using breathing sounds. Telemedicine and e-Health, 2020. 26(10): p. 1202–1205. 64. [64].Schuller, B.W., et al., COVID-19 and Computer Audition: An Overview on What Speech & SoundAnalysis Could Contribute in theSARS-CoV-2 Corona Crisis. Frontiers in Digital Health, 2021. 3: p. 14. 65. [65].Ting, D.S.W., et al., Digital technology and COVID-19. Nature medicine, 2020. 26(4): p. 459–461. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41591-020-0824-5&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F11%2F14%2F2021.11.12.21266271.atom) 66. [66].Topol, E.J., Is my cough COVID-19? Lancet, 2020. 396(10266): p. 1874.