An interactive dashboard to track themes, development maturity, and global equity in clinical artificial intelligence research

Publication search and metadata collation

A web-scraping application was produced using Entrez API⁹ to obtain titles, abstracts, metadata, and Medical Subject Heading (MeSH) terms from MEDLINE, for publications under broad AI search terms. From metadata we extracted publication date, journal, authors, affiliations, and derived geographical location of affiliations (see Supplementary Materials).

We used the following search terms for scraping publications from MEDLINE, based on search terms employed in previous systematic reviews of artificial intelligence: “““(((((((([“artificial intelligence”) OR (“deep learning”)) OR (“machine learning”)) OR (“neural net”)) OR (“transfer learning”)) OR (“supervised learning”)) OR (unsupervised learning))) OR (artificial intelligence[MeSH Terms])”““

Author affiliation geo-location

The location embedded in MEDLINE metadata refers to the journal country, rather than the country where research was conducted. We extracted geographic location of author affiliations using Python 3.8 geocoding libraries, including geopy (v2.2.0) for location, and folium (v0.12.1) for visualization.

BERT-PubMed

We employed a transfer learning approach, using state-of-the-art Bi-directional Encoder Representations from Transformers (BERT)¹⁰ NLP models with substantial pre-training on medical corpuses and academic abstracts (BERT-PubMed)¹¹.BERT, originally created by Google AI, enables bidirectional text representation to develop a deeper sense of language context. It is resilient to imbalanced datasets without need for additional methods for data augmentation¹². Classifier models were fine-tuned on manually labelled abstracts indexed on MEDLINE between 1998 and 2020.

Each classifier was trained using a 512-sequence length, increasing accuracy at the expensive of training cost. We used an AdamW optimizer. Training was performed using Tensorflow and Keras libraries on a local machine with an Nvidia GTX graphics card. Optimal epochs ranged from 3, for inclusion and maturity classifiers, to 5 for the multi-classifier. Training data and code hosted at: https://github.com/whizzlah/health_ai_training.

Inclusion classifier

In determining labelling criteria, we aimed to include all research papers that develop AI models for human healthcare, focusing on models that provide predictive, diagnostic, or quantitatively informative outputs that inform decision-making. We designed a guide question: “Does the proposed model output have a direct, actionable effect on patient care, by providing information to a healthcare provider, patient, or automated system?”. This excludes AI models for pre-processing images or data, or workflow assistance¹³. We aimed to exclude publications using non-human participants, reviews, and informal publication types.

BERT-PubMed was fine-tuned with a binary classification layer. Training was initiated on 4000 abstracts from 1998 to 2020 manually labelled for inclusion. Negligible numbers of positive samples were found prior to 1998. The training set was augmented actively by manually labelling additional abstracts classified with high uncertainty, until model performance reached satisfactory metrics. Evaluation was conducted on a test set (n=1034), and prospectively on abstracts from publications produced after pipeline completion (n=lO00, after September 2021). Classifier sensitivity against manual review was tested using a curated list of publications (n=446) from a recent systematic review of deep learning¹⁴ (Table 1).

Maturity classifier

Previous publications examine AI ‘maturity’ in two separate but parallel contexts^15,16. The first describes important methodological, data, and reporting characteristics within model training and testing that contribute to risk of bias and calibration. The second considers a high-level overview of stages of technological development, with previous research adapting National Aeronautics and Space Administration (NASA) technology readiness. However, these were designed for prototyping engineering devices and do not necessarily translate to development of heterogenous clinical AI models.

We consider technological development maturity using a novel framework specific to clinical AI, describing four stages prior to active real-world implementation (Figure 2). In summary, a “Math-into-Algorithm” stage represents development of novel algorithmic techniques, an “Algorithm-into-Model” stage represents the testing of model performance on datasets with ground truth labels, a “Model-into-Device” stage describes model testing against a non-AI, existing, gold standard (analogous to a comparative clinical study), and “Device-into-Practice” describes deployment for validation in a prospective, real-world environment. This framework has been employed in a recent review of AI in mechanical ventilation¹⁷.

We fine-tuned a second BERT-PubMed classifier to identify abstracts fulfilling at least a “Model-into-Device” stage of model development, initiated on 2500 manually labelled abstracts from 1998 to 2020 with active augmentation. The maturity classifier was evaluated on a test set (n=784), prospectively on abstracts from 2021 (n=2494, after September 2021), and output compared to curated publications from a systematic review of AI vs clinicians¹⁸ (n=83) (Table 2).

Study characteristic labelling and classification

We used a Named-Entity-Recognition (NER) NLP model (SparkNLP based on work by Chiu/Nichols) combined with a dictionary-based text recognition layer to discover and classify major entities expressed in abstracts, including clinical specialty, subspecialty or disease, type of algorithm used, and type of data input into models. For ease of interpretation, we did not consider Radiology as a ‘clinical specialty’, instead considering use of radiomics as an input feature. This avoids potential inconsistencies where, for example, an AI device for stroke diagnosis could either be classified under Radiology or Neurology. For the purposes of online deployment, manually validated NER labels were used to train a BERT-PubMed multi-classifier to label abstracts for major specialties, subspecialties, and data types. Training and testing were performed in a 3200:800 split dataset. Metrics and classes are shown in Table 3 and 4-

Dashboard deployment and hosting

The pipeline is deployed to the Google Cloud platform (https://cloud.google.com/) as Cloud Functions, triggered every 24-hours to discover new papers indexed in the preceding period. Due to the size of models and time required to label abstracts, the pipeline is split into two functions which share data via Google Cloud platform Pub/Sub. Google BigQuery (https://cloud.google.com/bigquery) is used to store scraped and labelled abstracts.

Deployed pipeline code hosted at: https://github.com/whizzlab/health_ai_online_pipeline

Performance

The final pipeline (Figure 1) performs a sequence of tasks, feeding into the dashboard. It uses broad AI-related search terms to identify and extract abstracts and metadata from MEDLINE, before performing geolocation on affiliations. It then runs through NLP tasks that (1) label abstracts for inclusion if they represent original research that develop clinical AI models; (2) identify research at a ‘mature’ development stage which describe either evaluation of an AI algorithm vs clinicians, or prospective real-world testing; (3) labels abstracts for research characteristics including clinical specialty, subspecialty, disease, input data type, and algorithm.

In prospective evaluation of ability to correctly classify publications indexed on MEDLINE after September 2021, the inclusion classifier (task 1) achieves an Fl of 0.96 and a Matthews correlation coefficient (MCC) of 0.94. The maturity classifier (task 2) achieves an Fl of 0.91 and MCC of 0.90. The multi-class classifier (task 3) for abstract characteristics achieves a macro-average Fl of 0.97 across classes. When evaluated against a recent systematic review of deep learning¹⁴, out of 446 publications identified by review authors, the pipeline correctly included 438 (98.2%).

Compared to a systematic review of comparative studies¹⁸, the pipeline correctly labelled 81 out of 83 (97.5%) for maturity. Detailed performance metrics are reported in Tables 1-4

Dashboard and global AI landscape

The interactive dashboard updates every 24 hours (https://aiforhealth.app), creating a live view of the current state of global AI. Dashboard datasets allow any cross-section of attributes to be extracted, compared, and analysed longitudinally.

To illustrate utility, we examined the entire research landscape before October 2021. Growth in clinical AI research became explainable by exponential growth functions from 2016, in total research (R2=0.999, p<0.001) and mature research (R2=0.998, p<0.001). We discovered 34178 examples of original AI research, with 1562 studies employing mature validation methods. Development and maturity heterogeneity across major themes over the past decade is illustrated in Figure 3. Lung, breast cancer, and retinopathy demonstrate substantial maturity relative to total research production, while cardiovascular, psychiatry, and infectious disease prediction lag behind. The distribution of data type usage across major subspecialties are shown in Figure 4 as heatmaps, demonstrating prevalence of mature validation using radiomics (and other imaging tasks) across all AI themes. Notably, only 1.3% of all research, and 0.6% of mature research, involved an author from a low to low-middle income country (per World Bank definitions), with 93.6% of such research published after 2016 (Figure 5).

• Download figure
• Open in new tab

Figure 3

Horizon charts of relative research output across major specialties and disease areas for all publications (blue) and mature publications (orange), for each year in the past decade. Colour density is used in addition to height to represent size, maximising use of available space. Design and interpretation as described by Heer et al²¹.

• Download figure
• Open in new tab

Figure 4

heatmap showing use of input features across subspecialty/disease areas in all (top,blue) and mature (bottom, orange) studies.

• Download figure
• Open in new tab

Figure 5

global distribution of clinical artificial intelligence research from 1998 to 2021 by first author, showing all publications (blue, top) and mature publications (orange, below). Where first author affiliation is not available, last author affiliation is used instead.