Accuracy of Combined Deep Learning Algorithms in Detecting Spontaneous Intracranial Hemorrhage on Emergent Head CT Scans

Ethical considerations

The local institutional review board of Helsinki University Hospital (HUH) approved the retrospective data collection and study design and granted a waiver for acquiring informed consents (HUS/365/2017; HUS/163/2019; HUS/190/2021). According to the Finnish legislation, no ethics committee approval is needed for retrospective studies that utilize registry or archive data. We gathered all imaging data for algorithm training from the HUH, which consists of 23 separate hospitals. All five Finnish university hospitals, including the HUH, are publicly funded non-profit organizations that provide tertiary health care services for all people living in Finland, regardless of socioeconomic status, insurance status, or race/ethnicity. Therefore, we believe that the collected HUH imaging data for algorithm training and validation is not inherently biased or deliberately discriminative. We conducted the study in line with the Declaration of Helsinki. The proofreading of the text was conducted using Chat GPT 4.0. STARD checklist for this study can be found as an online supplement.

Training data

Training dataset of the three new U-Nets consisted of 63, 50 and 67 head NCCT MPR- reformates (with 512 x 512 dimensions) for ICH, IVH, and SAH, respectively. All patients were imaged and treated at HUH. Segmentations were done using Philips IntelliSpace Discovery ¹⁷ and 3D Slicer ¹⁸. Image data and segmentation files were saved and used in NIfTI file format. We used Hounsfield unit (HU) threshold-based method to decrease human error and to increase reproducibility. We determined an acute blood threshold value for each bleeding type; 60-90 HUs for ICH and SAH and 50-90 HUs for IVH. We only segmented the bleeding type of interest for the base model training. To make a final agreement of the segmentation mask, all the segmented scans were reviewed by a cerebrovascular neurosurgeon and/or radiologist. After the review, segmentation masks were saved in a binary format, 0 meaning “no blood” and 1 meaning “blood”. We randomly chose 55 NCCT scans from the base model training NCCTs for the meta-model training dataset. For these 55 NCCTs we segmented all three bleeding types using the same HU thresholds mentioned above. The review process after segmentations remained the same. Eventually, the segmentation masks were saved also in a binary format and each segmentation in its own channel. The final segmentation files for the metamodel training included three channels (one for each bleeding type).

Model architectures and training

The trained three U-Nets were two-dimensional (2D) and had five convolution blocks with residual connections both in the encoder and decoder parts. Dropout and batch normalization layers were also used. Down sampling was done using a 2D convolutional layer with a kernel size of (2,2) and stride of (2,2). Dice loss was used as training loss for ICH and SAH U-Nets. For the IVH U-Net, we used a custom loss function where focal loss and dice loss were combined. Image data for ICH and SAH U-Nets were first clipped between -500 and 500 HUs and then normalized between 0 and 1. Image data for IVH U-Net was clipped between 0 and 200 HUs and then normalized between 0 and 1. Image data and segmentation masks were rotated 90, 180 and 270 degrees as an augmentation method during the training. Before training, each of the datasets were randomly split into training and validation set using 80:20 split ratio with 20% of the data reserved for validation purposes during training process. Training was done in Microsoft Azure using NVIDIA Tesla V100 graphics processing unit (GPU) with a batch size of 64, kernel size of (3,3) and dropout rate of 0.2. We used an Adam optimizer with a learning rate of 0,001. The training was set to last 200 epochs and the U-Net with the lowest validation loss value was selected for the final use.

We trained the metamodel after training the base U-Nets. The metamodel included a batch normalization layer and two 2D convolutional layers. The input of the metamodel is a 2D image with 5 channels (original CT slice and predictions from the four base U-Nets). The image channel was clipped between –500 and 500 HUs and normalized between 0 and 1. During the training, image data and corresponding segmentation masks were rotated 90, 180 and 270 degrees as an augmentation method. Similar 80:20 split ratio was used for the metamodel training dataset. Training was done in Microsoft Azure using V100 GPU with batch size of 64, kernel size of 3. The training was set to last 100 epochs. Learning rate of 0.003 and Adam optimizer were used. We used Dice loss as a loss function, and the model with the lowest validation loss was selected for the final use. The model outputs semantic segmentation, i.e. pixel-wise probability of the presence of a hemorrhage in a single NCCT slice.

For the inference pipeline, the combined U-Net solution included a reshaping of image data if the pixel data differed from 512 x 512. After resizing the image, the data was first sent to the base U-Nets with the corresponding normalizations. The base U-Net predictions were saved for later post processing steps. The predictions from the base U-Nets and the original imaging data (with clipping between -500 and 500 HU and normalization) were then directed to the metamodel for the final prediction. We used TensorFlow 2.2.0 to build and train the U-Nets and the metamodel.

Adaptive post-processing steps

To reduce the number of falsely predicted positive findings, we implemented post processing steps. First, we included only predictions in which segmented pixels (i.e. predicted blood) formed a cluster of a minimum 10 pixels in a single slice. A voting support from the base U- Nets was used to direct the slice further for a test time augmentation (TTA) step, or to push the slice forward without TTA. In the voting phase, all predictions from the base U-Nets were summed and compared against the metamodel predictions. If there was an overlap between the metamodel prediction and majority of the base U-Net predictions, the slice was classified as a strong positive. TTA was applied as a post processing step only if the slice was positive, but the prediction was not classified as a strong positive, as described above. By default, the solution made the predictions for input image data without any augmentations. In the TTA step, the image data was flipped both horizontally and vertically. The augmented images were then analyzed, and the predictions were then spatially reversed to match the predictions of the original data. All predictions were then summed and divided by 4. A removal of pixel clusters smaller than 10 pixels was also done after the TTA step. In the final step of the post processing, the positive prediction clusters were combined with the predictions of the base U- Nets, and if the combined cluster size exceeded 125 pixels, the segmentation was classified as a positive prediction. Of note, one 512 x 512 head NCCT slice contains 262 144 pixels. Examples of DL solution’s output is presented in figure 1.

• Download figure
• Open in new tab

Figure 1 True Positive Detections of Hemorrhage and Segmentations

True positive detections and corresponding segmentations of A) ICH, B) IVH and C) SAH. The left sided image in each panel shows the original CT scan slice and right sided image displays the DL solution’s detection of hemorrhage in the slice (purple color showing detected hemorrhage).

Real-world validation dataset

To simulate a real-world emergency imaging setting, we collected a retrospective dataset from 10 different hospitals in the HUH catchment area. These hospitals offer various levels of emergency care, ranging from primary care without neurosurgical services to tertiary care with such services. Together these 10 hospitals cover a catchment area of over 1,700,000 inhabitants in Southern Finland. The validation dataset consisted of all consecutive emergency head NCCTs imaged between October 1^st and December 31^st 2021 and corresponding on-call radiology reports. In more detail, if NCCT scan of adult patients (18 years or older) was performed either with emergent or immediate priority, it was included in the validation dataset. We also collected patient reports of the corresponding emergency room visits and ambulance reports, when available. In addition to these, we gathered information about patient demographics (age, sex, date of birth).

The on-call radiology reports were considered as ground truths, and every report was scrutinized to identify whether on-call radiologists had reported a primary spontaneous intracranial hemorrhage in the scan or not. Primary spontaneous intracranial hemorrhages included non-traumatic aneurysmal and non-aneurysmal SAHs, non-traumatic deep and lobar ICHs, and non-traumatic IVHs. Secondary spontaneous intracranial hemorrhages included SAHs, ICHs or IVHs related to ischemic strokes and tumors, and were excluded from the analysis. Also scans with reported poor quality, wrong imaging protocols (for example contrast-enhanced imaging and angiographies), and wrong reformatting (for example no axial series) were excluded from the dataset. We also excluded NCCT scans with inconclusive reporting of the presence of intracranial hemorrhages, and all follow-up scans. The dataset selection process is presented in figure 2. The dataset did not include head NCCTs from patients who were already admitted to a hospital ward before the NCCT was imaged. Also, if a single patient was imaged multiple times during different emergency clinic visits in the 3 months period, all first scans (excluding follow up scans) were included in the dataset.

• Download figure
• Open in new tab

Figure 2 Dataset Selection Flow Chart

We Reviewed all 8494 on-call report for the NCCTs. The final dataset included 118 NCCTs with spontaneous hemorrhage and 7679 NCCTs which were reported negative for hemorrhage in the on-call report.

The time of the symptom onset and the etiology of hemorrhages were evaluated on the basis of the patient and ambulance reports. If the etiology (traumatic vs. non-traumatic) of the hemorrhage could not be determined, it was classified uncertain. NCCTs with primary spontaneous intracranial hemorrhages were divided into five groups based on the time delay from symptom onset to imaging: 1) the exact time of symptom onset was mentioned in the patient or ambulance report, or in the imaging study referral, and the imaging was carried out within 12 hours from symptom onset, 2) the exact time of symptom onset was not clearly mentioned in the reports, but the imaging was done most likely within 12 hours from symptom onset, and 3) the exact time of symptom onset was not mentioned in the reports, but the imaging was performed most likely within 12- 24 hours from symptom onset or the exact time of symptom onset was mentioned, and imaging was carried out within 12 to 24 hours from symptom onset, 4) the symptom onset was between 24 hours to 7 days before the imaging or 5) the symptom onset was over 7 days before the imaging or the symptom onset remained unclear based on the available reports.

The head NCCT scans in which the on-call radiologist had reported hemorrhage were independently annotated on a slice-level by two study authors (JT and KV) for the presence of hemorrhage. Slices which included 5% or more missing image area (technical shortcoming related to head CT scanners) due to image reformatting were automatically considered negative for hemorrhages. The senior study authors solved any conflicts between the annotators either by removing annotations from the slice or accepting the slice annotations.

All imaging studies in the test dataset were in the DICOM format. Nvidia Tesla V100 GPU was also used for running the inference task in Microsoft Azure Machine learning studio.

Statistical analyses

We calculated the patient- and slice-level metrics by using the ground truth radiology reports and Python scripts tailored for these tasks. These reported metrics included sensitivity, specificity, false positive rate, negative predictive value and accuracy. We performed all statistical analyses with the Python package NumPy. We plotted the article figures using Matplotlib Python package.

Technical performance

Figure 3 depicts the time range taken for analyzing all slices in one head NCCT scan (median number of slices 53). The median time of analyzing all slices in a single axial 3 mm NCCT scan was 6.7 seconds (range from 4.2 to 33.1 seconds). The analyzing procedure consisted of multiple steps, such as reading the DICOM files, processing the imaging data, predicting the presence of a hemorrhage, post processing step, and saving the predictions in a NIfTI file.

• Download figure
• Open in new tab

Figure 3 Deep Learning Solution’s Run Times

Histogram of inference times for scans. Median run time for single scan was 6.7 seconds, 95 % of the scans could be analyzed within 10.6 seconds. The run time included processing of all slices in 3 mm axial reformat scan.

• Download figure
• Open in new tab

Figure 4 Deep Learning Solution’s Sensitivity

Sensitivity of the deep learning solution for detecting hemorrhages at different time point groups. The shaded are area in the plot represents the 95 % confidence interval.

Validation dataset

The final validation dataset included 118 NCCTs in which on-call radiologists reported spontaneous intracranial hemorrhage and 7679 NCCTs which were reported negative for hemorrhage by the on-call radiologist. The dataset included 4069 NCCTs of women and 3707 NCCTs of men (supplementary material – table 1). The median age of imaged patients was 71 years (range from 18 to 102 years) (supplementary material – table 1). The 7797 NCCTs were imaged using 12 different CT scanners from four vendors (supplementary material – table 2).

Of these, 59 were imaged within 12 hours, 19 presumably within 12 hours and 17 within 24 hours from symptom onset (Table 1). Fourteen intracranial hemorrhages were imaged between 24 hours and 7 days (Table 1). Nine intracranial hemorrhages had a delay more than 7 days, or the symptom onset was unclear (Table 1). Of the 59 early-imaged (within 12 hours) spontaneous intracranial hemorrhages, 32 contained only one type of hemorrhage, 22 two types and 6 three types. Overall, of 118 images in which on-call radiologists reported hemorrhage, 49 included only one type of hemorrhage, 38 included two hemorrhage types and 10 images included three types (supplementary material – table 3). The most common type was ICH (supplementary material – table 3). The dataset also included one acute subdural hemorrhage without known head trauma.

Case-level performance

The developed solution detected hemorrhages in 59 out of 59 patients (sensitivity 100.0 %) who were imaged within 12 hours from symptom onset (Table 1). The post processing step did not rule out any true positive cases. For 19 patients imaged most likely (not 100% certain) within 12 hours from symptom onset, the solution’s sensitivity was also 100 % (Table 1).

Additional 18 patients were imaged 12 to 24 hours from the symptom onset. The sensitivity of identifying acute hemorrhages among this group was 77.8% [95% confidence interval (CI), 56.3 - 96.6 %] (Table I). The four missed cases included two small ICHs (one in right internal capsule with maximum diameter of 6 mm and one in right pontine region with maximum diameter of 10 mm) and two local SAHs (supplementary material – figures 1.1- 1.4). For patients imaged between 24 hours to 7 days, the sensitivity of the solution was 71.4% [95% CI, 47.8 - 95.1 %]. The missed hemorrhage cases included one ICH (left sided thalamic with maximum diameter of 6 mm) and three local SAHs (supplementary material – figures 1.5-1.8). The sensitivity for hemorrhages that were imaged after 7 days from symptom onset or had unclear symptom onset time was 55.6% [95% CI, 23.1 - 88.0 %]. The missed hemorrhage cases included three ICHs (one resorbing left sided ICH near falx with maximum diameter of 5 mm hyperdense remnant remaining, one left sided frontal ICH with maximum diameter of 19 mm, and one right sided ICH in temporal region with maximum diameter of 29 mm) and one local SAH (supplementary material – figures 1.9-1.12). The sensitivity for all 118 cases of spontaneous primary intracranial hemorrhages was 89.8% [95% CI, 84.4 - 95.3 %].

Cases missed clinically but identified by solution

The solution identified five spontaneous intracranial hemorrhages that were not reported in the initial on-call reports. Of these five cases, one was ICH, two were SAHs, and one was IVH. In one imaging study, the on-call radiologist initially reported that the head NCCT showed a meningioma. Later the report was revised and stated that the finding had hyperdense regions which suggested that the finding was an intracranial bleeding instead of a meningioma. In other cases, the presence of hemorrhage was afterwards reported in additional reports by senior radiologists, or the diagnosis was done by the emergency room clinicians. All identified hemorrhages are presented in the supplementary material (supplementary material – figures 2.1 - 2.5)

Slice-level performance

The overall slice-level sensitivity was 69.3 % [95% CI, 67.3 - 71.3 %] and slice-level specificity 99.6 % [95% CI, 99.6 - 99.6 %]. The slice-level false positive rate for all images in the dataset was 0.4 % [95% CI, 0.4 - 0.4 %]. In more detail, of the 7679 NCCTs reported negative for hemorrhage by the on-call radiologists, the solution predicted falsely positive 1594 slices out of the 408 426 slices (Table 2). Most of the false positive pixel clusters pointed out blood in normal and highly vascularized anatomical structures, e.g. sagittal sinus, cerebellar tentorium, straight sinus, and falx cerebri (supplementary material – figures 3 and 4).