Risk-benefit analysis of the AstraZeneca COVID-19 vaccine in Australia using a Bayesian network modelling framework

2.1. Bayesian networks

BNs are acyclic graphical models that explicitly represent relationships between variables in terms of conditional probabilities [16]. Variables are represented by nodes, each categorized into several possible states (e.g., yes/no, high/low, age groups). Links (or edges) represent the relationships between nodes and point in the direction from a parent node (independent variable) to a child node (dependent variable) (Figure 1). Nodes are quantified using probability tables that define the probability of a node being in a given state, either based on some prior distribution (for nodes with no parent nodes) or conditional on the states of all parent nodes i.e., conditional probability tables (CPTs).

• Download figure
• Open in new tab

Figure 1.

Example Bayesian network showing three nodes (white boxes), links (arrows), and conditional probability table (CPT) for Die from Covid node (blue table). Data in figure are fictitious and for illustrative purposes only. The model assumes 90% vaccine effectiveness against death and even distribution of males or females in the population; these numbers are entered as priors because the nodes do not have parents. The model assumes case fatality rate of 20% in males and 10% in females. The CPT for Die from Covid shows that if the vaccine is effective, probability of dying is 0% for both genders. If the vaccine is not effective, the probability of dying is 20% for males and 10% and females. Fig 1a) shows the BN in its default state; if the vaccine was 90% effective, we expect an overall 1.5% chance of dying from COVID (e.g., in a population of 1000 people, the vaccine was not effective in 100 people. Of these 100 people, we estimate 10 deaths out of 50 males, and 5 deaths out of 50 females, i.e., total of 15 deaths out of 1000 people, or 1.5%). Fig 1b) shows an example scenario analysis of ‘what is the chance of a male dying if the vaccine was not effective?’ Selected states in each node are underlined, and the model updates the chance of dying from COVID to 20% under this scenario.

BN modelling frameworks offer many advantages relevant to the problem at hand. Firstly, BNs provide visual and transparent representation of relationships between variables and the probabilistic assumptions between each variable. The graphical structure enables users to understand how the system has been modelled, clearly shows the underlying assumptions, how model outputs were derived, and enables interactive scenario analysis. Secondly, CPTs can be derived from a range of data formats, allowing relevant data to be integrated from diverse sources including studies, reports, and expert opinion. Thirdly, the probabilistic framework of BNs is well suited to risk analysis where the uncertainty surrounding estimates can be tested through sensitivity analysis for a range of plausible inputs. Finally, model assumptions and CPTs can be updated easily as more information become available or as the situation evolves. Rapid adaptability is particularly important for modelling rapidly evolving situations such as outbreaks.

BNs have been used for a variety of COVID-19 models, including modelling transmission and outbreak response [17, 18], decision making [19], risk analysis [20, 21], risk assessment and contact tracing [22, 23], interpretation of SARs-CoV-2 test results [24], and predictive diagnosis [25]. However, to our knowledge, BNs have not yet been applied for risk-benefit analysis of COVID-19 vaccines.

2.2. Model design

Our BN model structure was based on the best available scientific evidence from multiple sources. Evidence was integrated using facilitated expert elicitation, with subject matter experts (JL, AB, KRS, AKE, CLL, Thrombosis & Haemostasis Society of Australia and New Zealand [THANZ]) working closely with experienced BN modellers (HJM, KM, CLL). The four subject matter experts included clinicians and researchers with expertise in infectious disease epidemiology, virology, thrombosis, general practice, and public health. The experts defined the scope of the model, key outcome variables of interest, and major predictor variables (inputs) that influenced the outcomes. Experts reviewed published literature and reports to determine whether there was sufficient evidence to include each predictor variable in the model. Predictors were linked to outcomes if there were sufficient evidence and data to quantify the conditional probabilities between them.

Based on the discussions between the experts and modellers, and evidence collected by the experts, a draft BN model structure was derived by the modellers, then reviewed and refined with the team over several iterations of facilitated discussions. To parameterize the model, questionnaires were developed to help experts structure the systematically collected data into CPTs. Variables were not linked if there were only weak quantitative relationships between them because the links would have little impact on model predictions. Data were updated as new evidence became available during the model development process from July-September 2021. External consistency was considered during model design so that predictions were consistent with other commonly used information sources such as the Australian Technical Advisory Group on Immunisation (ATAGI) and the Therapeutic Goods Administration (TGA), e.g., using the same age groups and definitions of low/medium/high community transmission intensity (equivalent to 0.05%/0.45%/5.76% of population infected over 6 months) [15].

2.3. Atypical severe blood clots

TTS is a rare and atypical form of blood clotting associated paradoxical thrombosis and low platelets. TTS is immunologically mediated, and can be trigged by medications (e.g., heparin) or vaccines (e.g., AZ vaccine) [26]. TTS is diagnosed on the basis of clinical criteria and includes atypical severe blood clots such as CVST and splanchnic vein thrombosis (SVT) [27, 28], which have also been reported in COVID-19 patients [29, 30]. CVST and SVT are distinct from the common types of venous thromboembolism such as deep vein thrombosis or pulmonary embolus, or other types of thrombocytopenia and have the highest risk of mortality in this group. SVT includes portal vein thrombosis (PVT), Budd Chiari Syndrome and mesenteric vein thrombosis. The data included in the model focused on PVT as it is the most common type of SVT, and has been specifically reported post-COVID, enabling a direct comparison. Our model therefore focused on comparing the risk of AZ vaccine-associated TTS with the risk of CVST and PVT in COVID-19 patients.

2.4. Data sources

While the assumptions and model structure are defined by experts, the CPTs were based on empirical data. Experts compiled evidence from peer-reviewed literature, government websites and reports, and through discussion with external clinical experts (e.g., haematologists regarding the evidence for AZ vaccine-associated TTS, and background rates of CVST and PVT). Official data from Australian authorities were used wherever possible (e.g., local data on AZ vaccine-associated TTS). Otherwise, data were obtained from other robust and publicly available sources (e.g., background rates of CVST and PVT). Data from different sources were harmonised for the CPTs, e.g., summarising data to match the age groups used in the BN. Data analyses were conducted for some variables to convert them into probabilities for the CPTs, e.g., converting incidence of COVID-19 into probability of infection over 6 months for the Intensity of local transmission node. Table 1 [31-40] and Appendix A provide a summary of data sources, model assumptions, and rationale.

The model includes default prior distributions for age group (using age distribution of Australia), gender (50% male, 50% female), SARS CoV-2 variants (90% delta, 10% alpha/ancestral virus), and vaccine coverage (70% of population received first dose, 35% received two doses). The prior distributions do not affect the results of scenario analyses, e.g., if delta was selected in the variants node, the model outputs relate to delta only regardless of the prior distribution of variants entered into the model. Prior distributions can be changed to model other scenarios, e.g., different distributions of variants, and different levels of vaccine coverage.

2.5. Model validation

The final model was assessed by subject matter experts and the modelling team together, to determine if the structure, variables, and assumptions reflected current knowledge and evidence. The model’s predictive ability was not validated against a dataset because model parameters were not learned from data. Rather, we validated each section of the model by defining a range of scenarios and manually assessed whether model outputs were consistent with our assumptions. Manual quantitative validations were conducted by a subject matter expert/BN modeller (CLL), a mathematician/statistician/BN modeller (KM) and a biostatistician (MW). Scenarios used for manual validations are provided in Appendix B. All authors evaluated the biological plausibility of estimates, e.g., for COVID-19 patients, the probability of dying from COVID-19 related atypical blood clots should be lower than the total probability of dying from COVID-19.

2.6. Sensitivity analysis

To determine how often model assumptions need to be updated, sensitivity analysis was conducted for three predictor variables that were considered most likely to change over time. We examined actual changes in the reported incidence of AZ vaccine-associated TTS cases and deaths in Australia in August-September 2021. We also examined i) plausible differences in age-specific CFR for COVID-19 in Australia (e.g., if higher CFR with new variants or because of an overwhelmed health system, and ii) plausible reductions in vaccine effectiveness against symptomatic COVID-19 infection and death.

3.1. Model description

The final BN structure (Figure 2) synthesises the assumptions regarding probabilistic relationships between variables and outcomes. Table 2 provides a summary of all nodes (n1 to n20) and their parent/child relationships. The model was designed to predict five outcomes:

• Download figure
• Open in new tab

Figure 2.

Bayesian network for assessing risks versus benefits of the AstraZeneca COVID-19 vaccine.

1. Probability of dying from AZ vaccine-associated TTS (n6) – depending on age (n2), 1^st or 2^nd dose of AZ vaccine (n1);

2. Background probability of deaths from CVST and PVT (in those who have not had AZ vaccine or COVID-19 infection) (n7, n8). Estimates were converted to probability of events over 6 weeks to enable comparison with the probability of vaccine-associated TTS, which generally occurs within 6 weeks of vaccination;

3. Probability of symptomatic COVID-19 infection – depending on age (n2), sex (n5), variant (n3), intensity of community transmission (n4), vaccine effectiveness against symptomatic infection (n9);

4. Probability of dying from COVID-19 (n18) – depending on age (n2), sex (n5), variant (n3), intensity of community transmission (n4), vaccine effectiveness against symptomatic infection (n9), vaccine effectiveness against death (n10); and

5. Probability of CVST and PVT deaths (n19, n20) related to COVID-19 infection – depending on sex (n5).

The BN includes five input nodes (orange) that could be used for scenario analysis based on AZ vaccine doses (n1), age (n2), sex (n5), variant (n3) and intensity of community transmission (n4). Community transmission scenarios were converted to probability of infection over six months to more readily compare the risks versus benefits of vaccination, which is expected to be effective for at least six months. Scenarios were selected based on ATAGI definitions of low/medium/high risk [15], other scenarios that users could relate to (e.g., 1000 cases/day in NSW), and general scenarios that could be adapted to any setting (e.g., 1% chance of infection over 6 months). The model includes nine intermediate nodes (yellow) where modellers can readily update evidence on incidence of TTS (n6), background incidence of CVST and PVT (n7, n8), relative risk of infection based on age and variant (n11), vaccine effectiveness (n9, n10), and incidence and CFR of COVID-19 related CVST and PVT (n12, n13).

Two versions of the model were built using different definitions of the ‘AZ vaccine doses (n1)’ node:

• Version 1: AZ vaccine doses defined as no doses, 1^st dose, and 2^nd dose. This version was used to estimate the probability of vaccine-associated TTS with each dose of vaccine.

• Version 2: AZ vaccine doses defined as no doses, received only one dose, and received both doses. This version was used to estimate the probability of deaths in the population based on vaccine coverage rates.

3.2. Model validation

A model walk-through indicated that subject matter experts agreed the model structure matched their understanding of the problem space in terms of the relevant predictors and relationships drawn from the external data sources. Independent manual calculations by three authors (CLL, MW, KM) of probabilities of selected outcomes were consistent with model predictions (Appendix B).

A minor anomaly was detected in the model through assessment of biological plausibility. For COVID-19 cases in younger age groups (females <30 years and males <10 years), the estimated probability of dying from COVID-19 was lower than the probability of dying from COVID-19 related atypical severe blood clots. The reason for this anomaly is that data for COVID-19 CFR in Australia were used for the CPTs, and there have been no deaths to date in these age-sex subgroups (Table A3), while data on COVID-19 related atypical blood clots were extracted from a study where data were predominantly sourced from the USA and Europe, where high CFRs could have resulted from an overwhelmed health system during the outbreak. Although the discrepancies in probabilities were extremely low (probability of dying from COVID-19 related atypical blood clots <0.0002% higher than dying from COVID-19 itself), the anomaly highlights that the model predictions should be used as broad estimates rather than exact risks.

3.3. Comparison of estimated deaths prevented from COVID-19 under different scenarios of community transmission intensity and vaccine coverage, with estimated cases and deaths from AZ vaccine-associated TTS

Model version 2 was used to estimate COVID-19 deaths prevented over 6 months per million population if 70% had a first dose, and 35% had two doses. Figures 3a, 3b, and 3c show the estimated deaths prevented for each age group under different levels of community transmission:

• Download figure
• Open in new tab

Figure 3.

Estimated COVID-19 deaths prevented over 6 months per million population of each age group if 70% had first dose, and 35% had two doses of AZ vaccine under a) low, b) medium, and c) high levels of community transmission; and d) estimated deaths from AZ vaccine-associated TTS if 70% of the population had first dose, and 35% had two doses. (Note the large variations in scale in y-axes between each graph).

• Low transmission (Figure 3a), similar to 1^st wave in Australia in 2020, equivalent to 0.05% of population infected over 6 months;

• Medium transmission (Figure 3b), similar to 2^nd wave in VIC in 2020, equivalent to 0.45% of population infected over 6 months; and

• High transmission (Figure 3c), similar to Europe in January 2021, equivalent to 5.76% of population infected over 6 months.

The model shows that for a million people aged ≥70 years where 70% have had first dose and 35% two doses, an estimated 25 deaths would be prevented under low transmission (Fig 3a) versus >3000 deaths prevented under high transmission (Fig 3c), with <1 expected death from TTS (Fig 3d). In a million people aged 60-69 years with the same vaccine coverage, the model estimates two deaths prevented under low transmission, and 260 deaths prevented under high transmission, with <1 expected death from TTS. In contrast, for a million 20-29 year-olds with the same vaccine coverage, <0.1 deaths would be prevented under low transmission, ∼9 deaths prevented under high transmission, with <1 expected death from TTS. Details on calculations are provided in Appendix C.

3.4. Comparison of risk of AZ vaccine-associated TTS with risk of atypical blood clots (CVST and PVT) in COVID-19 infected patients

Up to 25/8/2021 in Australia, age-specific incidence of AZ vaccine-associated TTS cases ranged from 16 to 27 per million first doses (Table 1), with an overall CFR of ∼5% (6 deaths out of 115 cases). Based on annual background rates of CVST and PVT reported by Kristoffersen et al. [27] and Ageno et al. [28], our model estimated 6-week incidence of 0.38 (age <20 years) to 2.69 (age ≥70 years) cases per million, and overall CFR ranging from 7.0% to 21.6% from youngest to oldest age groups (Table A7). The background CFR for CVST and PVT (combined) range from 0.03 (age <20 years) to 0.58 (age ≥70 years) per million over 6 weeks, compared to CFR of 0.83 to 1.40 per million first doses of AZ vaccine depending on age (Table 3).

Model assumptions on the incidence and CFR of CVST and PVT in COVID-19 patients were obtained from a retrospective cohort study using linked electronic health records predominantly from the USA and Europe [30] because there were insufficient data from Australia. Model version 1 estimated that overall fatality from atypical severe blood clots (CVST and PVT combined) in COVID-19 patients were 51.1 and 37.3 per 100,000 in males and females, respectively. Figure 4 shows that the probability of developing atypical blood clots in COVID-19 patients was 14-28 times more likely than developing TTS after the first dose of the AZ vaccine, depending on age group and sex (Figure 4, dashed lines). The probability of dying from COVID-19 infection-related atypical blood clots in COVID-19 patients was 58-126 times more likely than dying from TTS after the first dose of AZ vaccine, again depending on age group and sex (Figure 4, solid lines).

• Download figure
• Open in new tab

Figure 4.

Number of times more likely to develop and die from atypical blood clots (CVST and PVT) from COVID-19 infection than AZ vaccine-associated TTS, by age group and sex.

3.5. Sensitivity analysis

Age-specific CFR for COVID-19 in Australia

By August 2021, CFR for COVID-19 in Australia was 1.84% in males and 1.94% in females. In those aged under 50 years, CFR was <0.1% in both sexes (Appendix A). If there was one extra death in each age-sex subgroup during this time, CFR would have increased the most in 60-69 year-olds and the least in 20-29 year-olds, but by less than 0.06% in any subgroup (Figure 5). The model was most sensitive to changes in the 60-69 year-old age group because of the small case numbers (small denominator). If there were five extra deaths in each age-sex subgroup, CFR would have increased by less than 0.3% in any subgroup. Therefore, our model was not very sensitive to minor changes in number of reported deaths, and model estimates of deaths per million would have differed by less than 0.06% or 0.3% if there were one or five extra deaths in any age-sex subgroup, respectively.

• Download figure
• Open in new tab

Figure 5.

Change in age-sex specific case fatality rate for COVID-19 if one or five extra deaths in each subgroup: a) CFR in males, b) CFR in females, c) absolute difference in CFR if one or five extra deaths. (Based on COVID-19 cases and deaths in Australia by 31/8/2021).