Detecting CTP Truncation Artifacts in Acute Stroke Imaging from the Arterial Input and the Vascular Output Functions

Data

The ISLES’18 dataset is used for our experiments Cereda et al. (2016); Hakim et al. (2021). The database is multi-center and multi-scanner and includes 156 CTP scans obtained from 103 acute stroke patients. For our experiments, we have used the preprocessed scans from the ISLES 2018 challenge (http://www.isles-challenge.org/). The CTP volumes have been motion corrected, coregistered and spatio-temporally resampled (256 × 256 matrix, 1 volume per second). A full dataset description can be found in Cereda et al. (2016).

Simulating Shorter CTP Scans

We simulate shorter CTP scan durations by repeatedly discarding a 1 second timepoint from the end of the series until reaching the 10 first seconds of it. Note that the number of truncated simulated series varies from scan to scan, depending on its original total duration.

CTP Post-processing

Each truncated CTP series is analyzed using a research version of icobrain cva 1.4.1 (icometrix, Leuven, Belgium), an FDA-cleared and CE-marked software for acute stroke CTP post-processing. Each truncated series is processed using experts’ manually annotated vascular functions available in de la Rosa et al. (2021). Please note that the manual AIF/VOF does not change location for all shorter versions of a same scan. The vascular functions from each truncated scan are retained for the subsequent experiments.

Perfusion maps (Tmax, CBF, cerebral blood volume and mean transit time) are obtained through delay-invariant singular value decomposition deconvolution. Absolute and relative CBF maps are computed, where the relative rCBF map is obtained after normalization of the absolute one using mean control tissue values. Control tissue is defined by the software as Tmax < 6s Lin et al. (2016). Quantification of the hypoperfused and core lesion volumes is automatically obtained by the software using Tmax > 6s Lin et al. (2016) and rCBF < 0.38 (within the hypoperfused tissue area), respectively. The used rCBF cutoff (which is set in the software just for the purpose of these experiments) has been identified as optimal for the ISLES’18 dataset Cereda et al. (2016).

Defining Truncation Artifacts

In order to label each shorter scan version as reliable or unreliable (i.e., considerably suffering from truncation artifacts), we first check that the original unshortened scan does not already suffer from truncation artifacts. As such, scans are labeled to be stable if truncation of the final 6 frames or less did not impact the computed volumes by more than 2.5 ml Kasasbeh et al. (2016); otherwise, scans are labelled as unstable ones. For our experiments, all unstable scans have been discarded from further analyses. Besides, scans without a hypoperfused lesion have also been excluded as their stability can not be guaranteed.

The truncated series from all stable scans are labelled as reliable if the corresponding hypoperfused and core volumes deviated <10% or < 5 ml from the untruncated volume estimates. Otherwise, the truncated scan (and all its shorter versions) are labelled as unreliable. Besides, for each CTP series, the optimal scan duration is defined as the shortest scan duration providing reliable volumes estimates. Figure 1 shows a stable CTP scan example with its corresponding reliability truncation labels.

• Download figure
• Open in new tab

Figure 1:

Reliable/unreliable lesion volumes computed at various scan durations. The arterial input function (AIF) and the vascular output function (VOF) are displayed as reference.

Machine Learning for CTP Truncation Detection

Machine learning algorithms have been widely used to assess the quality of medical images Menze et al. (2008); Kyathanahally et al. (2018); Wei et al. (2019). We explore different machine learning models that could detect unreliably truncated scans by solely using information extracted from the vascular functions. The benefits of using the AIF and VOF to detect truncation artifacts are two-fold. First, the perfusion curves are always available in this imaging modality. Second, as they cover the entire perfusion event (note that these curves represent the contrast concentration inlet and outlet to the brain), they contain rich information for the problem under study. Consequently, it is needed to extract meaningful perfusion features that are impacted by an insufficient scan duration and that are, also, predictive of the truncation artifacts. Those features should capture the perfusion phases of the contrast-agent wash-in and wash-out and, ideally, they should be unaltered by the different CTP protocols used in clinical routine.

Feature Extraction

All the explored machine learning algorithms are fed with the following 9 AIF/VOF derived features:

• Scan duration

• AIF/VOF time to the peak of the function (argmax{AIF}, argmax{VOF})

• The AIF/VOF coverage, defined as the time difference between the peak of a signal and the scan duration:

  • ∗ AIF_coverage = scan duration - argmax{AIF}

  • ∗ VOF_coverage = scan duration - argmax{VOF}

• AIF/VOF upward and downward contrast increase

  • ∗ AIF_UCI = AIF_{t=argmax{AIF}} - AIF_t=0

  • ∗ AIF_DCI = AIF_{t=argmax{AIF}} - AIF_{t=scan duration}

  • ∗ VOF_UCI = VOF_{t=argmax{VOF}} - VOF_t=0

  • ∗ VOF_DCI = VOF_{t=argmax{VOF}} - VOF_{t=scan duration}

All features are visually represented in Fig 2.

• Download figure
• Open in new tab

Figure 2:

AIF and VOF derived features used to feed the machine learning algorithms. AIF: arterial input function; VOF: venous output function; HU: Hounsfield units; UCI: upward contrast increase; DCI: downward contrast increase.

Classifiers & Model Fitting

We train six statistical/machine learning classifiers with the aim of detecting reliable and unreliable truncated scans. The trained models make use of linear or non-linear decision functions and are: i) random forests, ii) multivariate logistic-regression, iii) support vector machines with linear kernel, iv) support vector machines with radial basis kernel, v) Adaptive boosting (aka, Adaboost Freund and Schapire (1997)) and vi) Gradient boosting Friedman (2001). All models are trained using the scikit-learn python library Pedregosa et al. (2011).

Data Augmentation

We augment our training dataset by generating synthetic samples in order to: i) compensate for the class imbalance between reliable and unreliable truncation samples and ii) model variable pre-contrast agent duration and variable contrast increases of the perfusion curves. Note that a different timing in the contrast bolus arrival alters the CTP scan duration but does not alter the presence of truncation artifacts. Likewise, the AIF and VOF contrast increase depends on the contrast agent iodine concentration. However, as the deconvolution algorithm is independent from the AIF/VOF absolute amplitudes, a variable vascular contrast increase does not alter the presence of truncation artifacts.

Balanced-class training sets are obtained using K-means SMOTE Last et al. (2017), a variation of the original synthetic minority oversampling technique Chawla et al. (2002), using the implementation in the Imbalanced-learn python library Lemaître et al. (2017). Simulation of contrast injection protocol variations is conducted by perfusion-specific data augmentation as similarly done in Robben and Suetens (2018); de la Rosa et al. (2021). Uniform distributions are used to randomly modify the pre-contrast agent duration and vascular contrast increases. When simulating variable pre-contrast duration, pre-contrast timing dependent features are increased or decreased by the same random factor (argmax{AIF}, argmax{VOF} and scan duration). For modelling variable contrast increases, the features AIF_UCI, AIF_DCI, VOF_UCI and VOF_DCI are scaled by a random factor.

Experiments

We perform a 5-fold cross-validation experiment using an 80-20% train-test data split. The data splitting is conducted at the scans level, assuring that i) all untruncated and truncated versions of a same scan belong to the same fold and ii) the same data-splits are used to fit all the considered models. Only the training data is used parametrise the models and to select the classifiers’ operating point. Truncation predictions are later inferred over the unseen test data.

Besides, we compare the machine learning models against a baseline classifier which solely uses the scan duration as discriminant-rule. The classifier g operates as follows: with θ a scan duration cutoff. This baseline is motivated by the CTP guidelines, which only consider the duration of a scan to avoid truncation artifacts Christensen and Lansberg (2019). Specifically, these guidelines suggest a cutoff of θ = 60 seconds in Equation 1. For our experiments, we evaluate the baseline classifiers’ performance at θ = [30, 40, 50, 60] s.

In order to understand the relevance of the AIF-VOF extracted features to discriminate reliably and unreliably truncated scans, we conduct a bootstrapping experiment by resampling 1,000 times the original database. In each iteration, a sample was drawn with replacement and was used to fit a classifier as described in Section Classifiers & Model Fitting. The relative feature importance is measured as defined in Friedman (2001) for decision tree ensembles. Briefly, the feature importance is calculated at the classifiers’ tree level as the impurity decay across all the nodes where that feature was used to create a split Kazemitabar et al. (2017). The final feature importance is computed as the average feature importance over all the considered trees. The mean and standard deviation feature importance for all features are reported. The chosen classifier for this experiment is the best performing one in terms of precision-recall AUC.

Performance evaluation

The mean, standard deviation, 5th-95th percentiles and minimum and maximum of the scan duration and the optimal scan duration are reported for the entire dataset. The different algorithms’ performance are evaluated by conducting receiver-operating-characteristic (ROC) and precision-recall (PR) analysis. The area under the ROC and precision-recall curves are used as general classifier performance metrics. Besides, we measure the binary classification performance at the operating point closest to an ideal classifier with precision = recall = 1. The operating point is chosen from the fitted classifier as , with t different classifier thresholds. Performance is measured in terms of precision , recall and F1-score , where acronyms represent TP: true positives, TN: true negatives, FP: false positives and FN: false negatives. The same binary classification metrics are reported for our baseline scan duration classifier, by making use of cutoffs θ = [30, 40, 50, 60] s. For these defined metrics, an unreliable truncation sample is considered as positive and a reliable truncation sample as negative.

Truncation Artifacts Detection

Figure 3 shows the ROC and precision-recall curves obtained with the different classifiers when differentiating reliable from unreliable truncated acquisitions. Overall it can be seen that classifiers yielded a similar high performance for both the considered metrics. The gradient boosting slightly outperformed the remaining classifiers with an ROC-AUC of 0.964 and a precision-recall AUC of 0.959. It is also worth pointing out that the multivariate logistic regression classifier obtained a lower performance than the baseline classifier, which solely uses the scan duration as input feature.

• Download figure
• Open in new tab

Figure 3:

Receiver operating characteristic (left) and precision-recall (right) curves. AUC: area under the curve; SVM_linear: support-vector machine with linear kernel; SVM_rbf: support-vector machine with radial basis function kernel; RF: Random forests; LR: Logistic-regression; Adaboost: Adaptive boosting; Gradboost: Gradient boosting.

When assessing the classifiers capability for detecting truncation artifacts at the chosen operating point, the results of Table 2 are obtained. Machine learning models yielded very similar performance for the considered metrics and have considerably outperformed the baseline classifier g(scan duration, θ = 60s). It is also worth noting that models with lower ROC and precision-recall AUCs (such as SVM_linear and SVM_rbf, Fig. 3) have provided slightly better results at the operating point compared to the Gradient boosting classifier. The highest F1-score for detecting unreliable perfusion volumes is achieved with a support vector machine with radial-basis-function kernel (F1-Score=0.913). In Fig. 4 we illustrate, for the outperforming classifier in terms of ROC and precision-recall AUC, the distribution of the predicted samples in terms of their scan duration to optimal scan duration difference (Fig. 4). It can be appreciated that all the mis-classifications are bounded in an approximate interval of ± 15 s. Thus, within this temporal window the classifier struggled the most to correctly detect unreliable perfusion volumes and, outside this temporal window, the classifier correctly predicted all samples. Besides, as expected, the closer a scan duration is to its optimal scan duration (i.e, the scan duration approximates to the inflexion point where unreliable samples become reliable), the harder for the model is to correctly classify a sample. Nonetheless, the absolute frequency of incorrect classifications is always much smaller than the absolute frequency of correct ones, independently of a scan duration’s closeness to its optimal scan duration.

• Download figure
• Open in new tab

Figure 4:

Histogram showing the difference between optimal scan duration and the scan duration for each predicted sample using a Gradient boosting classifier. Samples are grouped by their prediction status (correct/incorrect). Correctly predicted samples comprise true positives and true negatives. Incorrectly predicted samples comprise false positives and false negatives.

For the baseline classifier g, the highest detection performance is obtained at a cutoff θ =30 s (F1-score = 0.844). When using the clinical standard cutoff θ=60 s, the baseline classifier showed the maximal recall of 100% for detecting unreliably truncated scans. These results are expected as the optimal scan duration for the ISLES’18 dataset have much lower values than 60 s (Table 1). Although g(scan duration, θ=60 s) is extremely efficient at detecting truncation artifacts, it does it at the expense of generating many false positives (low precision of 0.469 and a low F1-Score of 0.638, Table 2). The ROC and precision-recall operating points at θ=60 s are shown in Figure 3. It can be appreciated that this operating point falls on the boundaries of the classifiers’ ROC and precision-recall curves. On one hand, a 60s scan duration could be a safe recommendation under fixed acquisition and post-processing considerations (as deconvolution type, injection and acquisition protocol) to avoid truncation artifacts, though it could still expose patients to unnecessary ionizing radiation when these considerations do not hold. On the other hand, when performing quality analysis on already acquired CTP scans, the 60 s scan duration is a very poor criterion to identify truncation artifacts as it does not consider any of the truncation confounders listed above. Our experiments show that machine learning models fed with perfusion-derived features can unveil acquisitions affected by truncation and reliably detect erroneous perfusion measurements. The proposed features are simple, robust to extract even in low quality acquisitions and independent (except for the scan duration) from the contrast injection and CTP acquisition protocols.

Importance of the AIF and VOF Features

Figure 5 summarizes the different features’ relevance obtained when fitting 1,000 Gradient boosting classifiers in a resampling with replacement bootstrapping fashion. The AIF_coverage shows to be the most crucial feature for detecting unreliable perfusion volumes due to truncated acquisitions. Besides, the VOF_DCI, VOF_coverage and AIF_UCI also result to be important features for the machine learning model. The large predictive value of the AIF_coverage and the VOF_coverage features can be related to their robustness to variable pre-contrast agent durations. The scan duration feature, instead, is affected by the CTP acquisition protocols and as such, shows less relevance for the fitted models. The contrast increase features AIF_UCI and VOF_DCI are useful for the task as they represent the beginning and ending of the agent delivery through the tissue, allowing the classifier to capture the characteristics of an entire or truncated perfusion event.

• Download figure
• Open in new tab

Figure 5:

Relative feature importance for 1,000 bootstraps with a Gradient boosting classifier. Bars (error-bars) represent mean (standard deviation). AIF: arterial input function; VOF: venous output function; UCI: upward contrast increase; DCI: downward contrast increase.

Based on these results we finally explore whether the most relevant feature, AIF_coverage, might have discriminant power to detect truncation artifacts. To this end, AIF_coverage is used instead of the scan duration to build a new classifier g^′(AIF_coverage, ρ) that operates as described in Eq. 1. While the scan duration based classifier g obtained a ROC and precision-recall AUCs of 0.940 and 0.933 respectively, the classifier g^′ using AIF_coverage as unique feature yielded 0.960 and 0.949 ROC and precision-recall AUCs respectively. These results evidence that AIF_coverage is a strong discriminant feature for detecting truncation artifacts. Besides, the close performance of g^′ to the results obtained with machine learning models (Fig. 3) show that these algorithms mostly predict outputs based on the AIF_coverage information and get some extra benefit from the remaining perfusion features.

Effect of the Data Augmentation

Finally we conduct an ablation study to assess the effect of the two used data augmentation approaches to train the machine learning models. For this experiment, a 5-fold cross validation scheme is followed by training a Gradient boosting classifier in three ways: i) With the original, un-augmented dataset, ii) Applying perfusion-specific data augmentation by modelling variable contrast bolus arrivals and variable AIF/VOF contrast increases and iii) By applying perfusion-specific data augmentation and class-balancing with K-means SMOTE Last et al. (2017). Results are summarized in Table 3. It can be seen that both the ROC and the precision-recall AUC improve when simultaneously using both types of data-augmentation to generate synthetic samples.

Limitations and Future Directions

There are some considerations about this research that should be cautiously taken. It is worth to mention that our conclusions only hold for CTP analysis using time-invariant singular value decomposition deconvolution. Other techniques used for perfusion analysis might behave differently under truncation scenarios. Still, the delay-invariant singular value decomposition deconvolution is the most widespread and used algorithm in software packages Fieselmann et al. (2011); Kudo et al. (2010); Vagal et al. (2019). Readers interested in the effect of CTP truncation over different parameter map estimation methods are referred to the work of Copen et al. (2015), as such inter-algorithm comparisons are out of the scope of this research. It is worth saying that while the devised models only hold for the ISLES’18 database characteristics and for the deconvolution algorithm used in this study, the extracted features are generalizable and allow the adaptation of these models to other deconvolution algorithms or imaging modalities (as perfusion MRI). In addition, the deployment of a truncation artifacts detection method in automatic CTP evaluation software is limited to the AIF/VOF selection performance. In this work, all the experiments have been conducted using manually annotated vascular functions. As such, failures in the CTP curves selection could produce a misleading truncation analysis using our proposed methodology. Nonetheless, recent approaches using dedicated artificial intelligence methods show efficacy and robustness to select vascular functions even under low quality CTP scenarios Winder et al. (2020); de la Rosa et al. (2021). Finally, future directions for this work might involve the machine-learning prediction of missing CTP time-points at the end of the series. As such, reconstructing the ending perfusion phase of the vascular functions could help improve the detection of truncation artifacts.