Clinical Uncertainty In Large Vessel Occlusion Ischemic Stroke (CULVO): An Intrarater And Interrater Agreement Study

Case Selection

All included studies were taken from an institutional database of patients undergoing in-house neuroimaging for symptoms of acute ischemic stroke. A team of clinical and research physicians reviewed our database to identify reported large vessel occlusions in patients being scanned for symptoms of stroke from January 2018 to August 2022. __ a dual trained stroke neurologist and neurointerventionalist reviewed all the flagged studies to determine eligibility using the following criteria: 1) > 18 years old, 2) stroke onset or last known well time between 6-24 hours prior to start of imaging (late-window stroke), 3) use of automated perfusion CT (RAPID, iSchemaView, Menlo Park, CA), 4) confirmed large vessel occlusion on CTA involving: first segment of the middle cerebral artery (M1), terminal portion of the internal carotid artery or tandem occlusion, 5) penumbra size of at least 15cc of perfusion CT, 6) mismatch ratio of at least 1.8, 7) National Institutes of Health Stroke Score (NIHSS) of 6 or more. For quality control, neuroimaging with unsatisfactory elements (artifacts, improper timing of contrast, or poor perfusion study quality) were excluded.

To minimize the paradoxes of Kappa statistics,^16,17 we included the full range of infarct core sizes: 10 cases with small core (0-49 cc), 10 cases with medium core (50-100cc), 10 cases with large core (>100cc). We selected cases with the intention of assessing the reliability (reproducibility) of clinician judgments within an experimental environment that encompasses a comprehensive range of core infarct sizes and clinical characteristics.¹⁸

Rater Selection

We invited stroke neurologists and neurointerventionalists in Canada who work in EVT capable stroke centers with access to perfusion imaging. Clinical fellows with at least a year of fellowship were allowed to participate. Invitations were sent to the physicians through the Canadian Stroke Consortium and the Canadian Interventional Neuro Group mailing lists. The raters were classified as: stroke neurologists, neurointerventionalists (neuroradiologists and neurosurgeons) and dual-trained neurologists who are both stroke neurologists and neurointerventionalists.

Survey

Selected cases were collected using Microsoft Excel v16.72 and placed into the survey format on REDCap (Research Electronic Data Capture). ^19,20 Personalized links were emailed to the participants. The following case details were provided: age, location of occlusion, the NIHSS, time of onset, time of scan, non-perfusion CT neuroimaging (NCCT and CTA). The NCCT scans were shown in stroke window settings (window width of 35HU and window level of 35HU).^21–23 The same 30 cases were shown again with the same information but with perfusion CT neuroimaging. Overall, the participants encountered 60 cases comprised of the same 30 patients shown twice with and without perfusion imaging. The raters were not informed that these 60 cases were the same 30 patients shown twice. To test intra-rater agreement, a second round of survey was done at least 3 weeks from the first. The survey was pilot tested and revised based on feedback from 7 trainee physicians who did not meet the study’s official rater criteria.

For each case, the raters were asked to grade ASPECTS, the single-phase collateral score (collateral score 0: absence of vessels distal to the occlusion, 1: ≤ 50% but >0% collateral supply, 2: ≥ 50% but <100% collateral supply or 3: 100% collateral supply) and finally whether they would recommend EVT based on the available clinical and radiologic data.^24,25

Sample Size

We estimated, using kappaSize package in R version 4.2.0 (R Foundation for Statistical Computing, Vienna, Austria), assuming an anticipated kappa value of 0.6 (substantial), that at least 30 cases were necessary for the lower limit of a 95% one-sided confidence interval to remain above 0.45 between at least 6 raters, considering an anticipated prevalence of EVT recommendation of 0.5.²⁶ A rule of thumb for agreement studies with binary outcomes is to have at least 10 raters reviewing at least 30-50 patients.¹⁸ While more patients will enable us to provide an even wider spectrum of patients, we also want to avoid rater fatigue. For this study, we selected 60 cases with three questions each. We needed at least 6 raters based on the sample size computation but aimed for at least 10 to follow the current convention.

Agreement Statistics

The inter-rater and intra-rater agreement of the recommendation of EVT were assessed using Gwet’s AC1 (κ_G) reliability coefficient. AC1 stands for agreement coefficient, first order chance correction and is used for binary data. ²⁷ We derived the κ_G for binary (EVT recommendations) data, with 95% bias-corrected confidence intervals. The agreement categories were defined as poor (κ_G <0), slight (κ_G =0–0.20), fair (κ_G =0.21–0.40), moderate (κ_G =0.41– 0.60), substantial (κ_G =0.61–0.80) and excellent (κ_G >0.80) according to the Landis and Koch criteria.²⁸ Non-overlapping confidence intervals and level of agreement were considered for potential difference between KG values.²⁹ The following prespecified subgroup analyses of interrater reliability were conducted: rater specialty, rater experience, infarct core size and time since stoke onset. For intrarater reliability, we looked at rater specialties. We used the R statistical computing language (R Core Team, 2022) in the RStudio framework to compute for the K_G values using the irrCAC package. ³⁰ The κ_G statistic has been shown to be relatively resistant to the Kappa paradox. κ_G has a less severe correction for chance agreement compared to Cohen’s Kappa because it does not assume all observed ratings may yield an agreement by chance and it places greater emphasis on hard-to-score subjects. ^27,31

We had initially planned to perform interrater agreement analysis on the first round of the survey only. However, two new randomized controlled trials (ANGEL-ASPECT and SELECT 2) on EVT for large vessel occlusion stroke patients with large core infarcts were published during the study period.^9,10 We felt that these new trials might influence how raters make their decisions; thus, we also estimated the interrater reliability on the second round.

Ethics

The study was approved by the research ethics board of .

Consent from patients was waived. Survey participants’ consent was obtained before the survey, and reaffirmed after, to ensure their continued agreement despite the study’s inherent deceptive methodology.

Patients

A total of 3,144 cranial CT scans done for symptoms of stroke were reviewed. Of these, 912 were found to have intracranial vascular occlusion. After applying the inclusion criteria, 120 cranial CT scan of large vessel occlusion stroke patients were found eligible (eFigure 1). The final 30 were selected to balance out clinical characteristics and radiologic features.

eTable 1 summarizes the characteristics of the 30 large vessel occlusion stroke patients whose CT scans were selected for final inclusion in the study. The median age was 77.0 years (min, max: 39.0, 92.0), 46.7% were females, most of the occlusions (60%) were on the middle cerebral artery and 40% of the patients underwent treatment with EVT. The mean NIHSS was 17 (standard deviation: 7), the median officially reported ASPECTS was 5 (min, max: 0, 10) and the median core infarct size was 82cc (min, max: 0, 270).

Raters

Invitations were sent to the members of the two Canadian societies of physicians directly involved in EVT decision-making for large vessel occlusion stroke patients, the Canadian Interventional Neuro Group (31 members) and the Canadian Stroke Consortium (186 members). Thirty-four respondents began responding to the initial survey that opened on November 29, 2022 but 5 did not complete it, leaving 29 respondents available for analysis. Of the 29 that finished the first round of surveys, 11 (37.9%) were stroke neurologists, 11 (37.9%) were neurointerventionalists and 7 were dual trained. Ten (34.5%) were senior raters with 10 or more years of experience. All except one rater finished the second round which opened on January 19, 2023. (See eTable 2)

EVT Decisions

Involving 60 CT scans of large vessel occlusion stroke cases and 29 raters, a total of 1,740 decisions were rendered regarding the recommendation for or against thrombectomy. The characteristics associated with the decisions regarding treatment made by the raters are summarized in Table 1. Substantial discrepancies in median values and frequencies of recommendations favoring or against EVT are observed in the following: Alberta Stroke Program Early CT Score (ASPECTS) (3 versus 7), core size (125 cc versus 50cc), collateral scores, and National Institutes of Health Stroke Scale (NIHSS) scores (20 versus 14). The percentage of decisions made against (50.7%) or for (49.5%) EVT based on perfusion CT scan was comparable.

The mosaic plot in Figure 1 shows the distribution of EVT decisions for 870 cases (30 cases shown to 29 raters twice—non-perfusion CT neuroimaging and perfusion CT neuroimaging). The overall recommendation for EVT was 59.0% (513/870) without perfusion images and 57.8% (503/870) with perfusion images a difference of only 1.1%. The observed agreement for no EVT (30.9%, 269/870) and for EVT (47.7%, 415/870) was 78.6% (684/870). Among 870 cases studied, 11.3% (98 cases) recommended EVT based on non-perfusion CT but not on perfusion CT scans, while 10.1% (88 cases) advised against EVT on non-perfusion CT but recommended it on perfusion CT for the same patients. Despite the small difference 1.1% in total EVT recommendations, decisions changed in 21.4% of cases.

• Download figure
• Open in new tab

Figure 1. How does neuroimaging affect endovascular thrombectomy (EVT) decisions?

A mosaic plot with a contingency table overlay is utilized to display the distribution of 870 cases (30 patients x 29 raters) and the corresponding decisions for or against endovascular thrombectomy (EVT) between two imaging protocols. All percentages are based on the denominator of 870. All box sizes are proportional to the magnitude of decisions. For instance, the lower left box represents the 10.1% (88/270) of all decisions. In this box, EVT was not recommended on non-perfusion imaging but recommended on perfusion imaging. Conversely the upper right quadrant of the plot signifies cases where EVT was recommended based on non-perfusion neuroimaging but not on perfusion neuroimaging (11.3%, 98/270). Conversely, the lower left quadrant represents cases that were not recommended for EVT based on non-perfusion CT neuroimaging but were deemed appropriate for EVT following perfusion CT neuroimaging.

Inter-rater Agreement

The interrater (κ_G) agreement is similar with non-perfusion CT neuroimaging compared to perfusion CT neuroimaging (κ_G = 0.487; 95% CI 0.327,0.647 and κ_G =0.552; 95% CI 0.430,0.675). (See Figure 2) The confidence intervals overlap and both K_G signify moderate agreement.²⁸ Similarly the interrater agreement was also not significantly different between the neuroimaging protocols when calculated according to physician subspecialty and experience. Both the subgroups for dual trained physicians and small core infarcts achieved substantial agreement across both neuroimaging protocols. The level of agreement increased (moderate to substantial) from non-perfusion CT neuroimaging to perfusion CT neuroimaging in the subgroups of stroke onset of 6-12 hours CT (κ_G =0.502; 95% CI 0.237,0.767and κ_G =0.621; 95% CI 0.427,0.815) and large core infarcts (κ_G = 0.451; 95% CI 0.367,0.76 and κ_G =0.665; 95% CI 0.411,0.919). (See Table 2)

• Download figure
• Open in new tab

Figure 2. Interrater Agreement for Endovascular Thrombectomy Between Two Surveys.

Clustered bar graphs with error bars show Gwet’s AC1 (κ_G) statistic with 95% confidence interval for inter-rater agreement between neuroimaging modalities and two survey rounds. There is extensive confidence intervals overlap between neuroimaging modalities and time from stroke onset indicating no differences. The level of agreement is marked as moderate (lower black line) or substantial (higher gray line) above the threshold lines.

As we elaborated on earlier, we performed an unplanned inter-rater analysis of the second survey round. (See Figure 2) There was no significant increase or decrease in agreement between the first and second round in both neuroimaging protocols as confidence intervals overlapped and stayed in the moderate agreement classification. Even for the subgroup of large core infarcts, the agreement level stayed at moderate in the second round (κ_G = 0.443; 95% CI 0.081,0.805 and κ_G = 0.550; 95% CI 0.253,0.847).

Intrarater Reliability

Twenty-eight (out of 29) raters finished both surveys. Table 3 summarizes the intra-rater agreement of these 28 raters according to the type of neuroimaging reviewed for their decisions. Confidence intervals of all intra-rater agreements overlap between both neuroimaging protocols for each of the 28 raters signifying no significant difference as per our predefined criteria.

Seventeen of the 28 raters (60.7%) had a change in the classification of intrarater agreement from non-perfusion CT neuroimaging to perfusion CT neuroimaging. With the addition of automated CT perfusion images, 11/17 had improved by at least one category while 6/17 had worsened by at least one category. eFigure 2 demonstrates the distribution of K_G for intra-rater reliability of the 28 raters. It illustrates that while the K_G values remain in the same category (substantial), the median value for perfusion CT is higher than non-perfusion CT.

EVT Decisions

Even though the raters were looking at the same patient clinical data, NCCTs and CTAs, their decisions for EVT changed in one out of every 5 patients (21.4%, 186/870) with perfusion imaging and these changes occurred in both directions, 10.1% (88/870) for EVT and 11.3% (98/870) against EVT. Taken together these changes in decisions with perfusion CT neuroimaging only led to 1.1% (10/870) fewer recommendations for EVT. Tabulating EVT recommendations, we saw that decisions for EVT were more common for patients with better collaterals, higher ASPECTS, smaller infarcted cores. These are all intuitive as these factors all portend a better prognosis to large vessel occlusion stroke patients undergoing EVT.^25,32

There are several possible reasons for these changes in decisions based on the neuroimaging presented. For those that go from recommending EVT to recommending against it (non-perfusion CT neuroimaging to perfusion CT neuroimaging), this may be from seeing the core infarct quantified. While early ischemic changes may not be apparent on non-contrast computed tomography (NCCT), they may appear larger on automated perfusion images. This could be attributed to genuine ischemia or the manifestation of the ghost core phenomenon, wherein the core infarct appears larger on perfusion imaging than its actual size. ³³ (See Patient 7 in eFigure 3) For those that go from recommending against EVT to recommending it (non-perfusion CT neuroimaging to perfusion CT neuroimaging), the decision is likely based on seeing a large value for the penumbra on automated perfusion images or a smaller core to the ASPECTS they calculated. Despite the possibility that the frank hypodensity observed on non-contrast computed tomography (NCCT) is the only region affected by the arterial occlusion, automated perfusion images can reveal viable regions surrounding the core infarct that could potentially be salvaged. (See Patient 17 in eFigure 3)

Interrater Reliability

Interrater agreement (K_G) on the decision to recommend EVT for late window large vessel occlusion stroke patients was comparable whether raters viewed non-perfusion CT neuroimaging or perfusion CT neuroimaging. Both were classified as moderate agreement and overlapping confidence intervals. The same results, overlapping confidence intervals at the same level of agreement, were found in both the first and second rounds of the survey. These findings support our hypothesis that the addition of perfusion imaging does not alter the reliability of decisions made by physicians of different specialties on a wide spectrum of large vessel occlusion stroke patients. In contrast, a similar study showed overlapping confidence intervals with the addition of automated perfusion imaging but had an increase in agreement from moderate (K = 0.506) to substantial (K=0.704) with the addition of automated perfusion imaging.³⁴ This is likely accounted for by the differences in study design as the aforementioned study utilized only two raters and gave an “indecisive” option for EVT decisions.

Intrarater Reliability

The intrarater agreement statistics (κ_G) on the decision to recommend EVT for late window large vessel occlusion stroke patients for each rater had overlapping confidence intervals between non-perfusion CT neuroimaging or perfusion CT neuroimaging. The extremes of intra-rater agreement are striking: non-perfusion CT (min, max: 0.359, 0.817) and perfusion CT (min, max: 0.416, 896). We posit that this difference may be due to some reviewers already being familiar with the newly published randomized trials and some not.^10,11 In essence, the reviewers with lower intrarater agreement may have “disagreed” with themselves on account of the new evidence. While those with higher K_G may not have reviewed the evidence or do not yet subscribe to it.

Limitations

Our study has notable limitations, including the absence of a universally accepted method for comparing agreement statistics (KG) beyond magnitude documentation. To gauge similarity in KG values, we relied on overlapping confidence intervals and comparable levels of agreement. ²⁹ Additionally, the study’s applicability is restricted to Canadian-based raters, and it does not encompass other stroke imaging modalities. We also omitted detailed clinical information and did not conduct an analysis of clinical outcomes in relation to the chosen imaging modality. Finally, examining causality between baseline patient and rater variables, including the choice of neuroimaging modality, and the decision to recommend EVT is not within the study’s scope.