The effectiveness of eight nonpharmaceutical interventions against COVID-19 in 41 countries

Appendix A.1. Detailed model description

• Download figure
• Open in new tab

Figure A.6:

Model Overview. Purple nodes are observed. We describe the diagram from bottom to top: The effectiveness of NPI i is represented by α_i, which is independent of the country. On each day t, a country’s daily reproduction number R_{t, c} depends on the country’s basic reproduction number R_0,c and the active NPIs. The active NPIs are encoded by F_{i, t, c}, which is 1 if NPI i is active in country c at time t, and 0 otherwise. R_{t, c} is transformed into the daily growth rate g_{t, c} using the generation interval parameters, and subsequently is used to compute the new infections and that will turn into confirmed cases and deaths, respectively. Finally, the number of new confirmed cases C_{t, c} and deaths D_{t, c} is computed by a discrete convolution of with the respective delay distributions. Our model uses both death and case data: it splits all nodes above the daily growth rate g_{t, c} into separate branches for deaths and confirmed cases. We account for uncertainty in the generation interval, infection-to-case-confirmation delay and the infection-to-death delay by placing priors over the parameters of these distributions.

We construct a semi-mechanistic Bayesian hierarchical model, similar to Flaxman et al¹, but with several key extensions. First, we model both confirmed cases and deaths, allowing us to leverage significantly more data. Furthermore, we do not assume a specific infection fatality rate (IFR) since we do not aim to infer the total number of COVID-19 infections. Additionally, since epidemiological parameters are only known with uncertainty, we place priors over them, following recent best practice². Concretely, we place priors on the means and standard deviations/dispersions of the generation interval, the infectionto-confirmation delay, and the infection-to-death delay. These prior choices are detailed in Appendix A.3. The end of this section details further adaptations which allow us to minimise assumptions about testing, reporting, and the IFR. Code is available online here. For readers who wish to re-implement the model, a concise list of all equations is given in Appendix H.

We describe the model in Figure A.6 from bottom to top. The epidemic’s growth is determined by the time-and-country-specific (instantaneous) reproduction number R_t,c. It depends on: a) the basic reproduction number R_0,c without any NPIs active and b) the active NPIs. We place a prior distribution over R_0,c, reflecting the wide disagreement of regional estimates of R₀3. The mean R_0,c is 3.28, based on a meta-analysis⁴. We parameterize the effectiveness of NPI i, assumed to be same across countries and time, with α_i. Each NPI is assumed to have an independent multiplicative effect on R_t,c as follows: where ϕ_i,c,t = 1 means NPI i is active in country c on day t (ϕ_i,c,t = 0 otherwise), and I is the number of NPIs. We place an Asymmetric Laplace prior over α_i with scale parameter 10, asymmetry parameter 0.5 and location parameter 0. Our prior allows for (unbounded) positive and negative effects as we cannot a priori exclude the possibility that the introduction of an NPI increases R. However, our prior places 80% of its mass on positive effects, reflecting a belief that NPIs are much more likely to reduce R_t than not. This is a shrinkage prior, placing 50% of its mass on ‘small’ effectiveness (less than 10% change in R_t). All prior distributions are independent.

Appendix A.2. Testing, reporting, and infection fatality rates

Scaling all values of a time series by a constant does not change its growth rates. The model is therefore invariant to the scale of the observations and consequently to country-level differences in the IFR and the ascertainment rate (the proportion of infected people who are subsequently reported positive). For example, assume countries A and B differ only in their ascertainment rates. Then, our model will infer a difference in (Eq. (A.3)) but not in the growth rates g_t,c across A and B. Accordingly, the inferred NPI effectiveness will be identical.^c

In reality, a country’s ascertainment rate (and IFR) can also change over time. In principle, it is possible to distinguish changes in the ascertainment rate from the NPIs’ effects: decreasing the ascertainment rate decreases future cases C_t,c by a constant factor whereas the introduction of an NPI decreases them by a factor that grows exponentially over time.^d The noise term, exp (Eq. (A.3)), mimics changes in the ascertainment rate—noise at time τ affects all future cases—and allows for gradual, multiplicative changes in ascertainment.

Appendix A.3. Method for uncertainty over key epidemiological parameters

We account for uncertainty in the following delay distributions: the generation interval, the delay from infection to symptom onset (incubation period), the delay from symptom onset to case confirmation, and the delay from symptom onset to death.

Each distribution has an uncertain mean (Table A.3), whose prior distribution we take from a meta-analysis¹³ published shortly after our window of analysis. This helps account for possible heterogeneity between different populations, and therefore often finds higher uncertainty in the means than estimated in primary studies, while using more data. The meta-analysis reports mean values with 95% confidence intervals. We set normal priors over the mean delays, with mean equal to the reported mean in the meta-analysis and standard deviation set such that the prior 95% credible interval includes the 95% confidence intervals from the meta-analysis. For example, the meta-analysis reports an expected mean delay from symptoms to death of 16.71 days with a 95% credible interval ranging from 15.37 to 18.17. We thus assume that the prior is normal with µ = 16.71 and σ = 0.75, which ensures that its 95% credible interval ranges from 15.25 to 18.17, strictly including the interval reported in the meta analysis to be conservative. Priors are truncated at zero.

Furthermore, we account for the uncertainty in the standard deviation of these delay distributions by placing normal priors based on primary studies, following the same method as above (Table A.4). For the standard deviation of the generation interval, we use patient data from Feretti et al.¹⁴ and reproduce their method, fitting a Gamma distribution to the generation time.

Finally, the distributional forms of the delay distributions are taken from the above primary studies.

The delay from infection to case confirmation is the sum of the incubation period and the symptom-onset-to-case-confirmation delay. Similarly, the delay from infection to death is the sum of the incubation period and the symptom-onset-to-death delay. However, for computational tractability, we model the total delay distributions, converting the priors described above to priors over the total delays. We assume that the total delays follow negative binomial distributions, which are computationally tractable and empirically provide a close fit to both sum distributions. We place normal priors over the mean and dispersion parameters of these total delay distributions by bootstrapping. For example, we compute priors for the total infection-to-case-confirmation delay distribution by sampling N_bootstrap = 250 incubation periods and symptom-onset-to-case-confirmation distributions. For each sampled pair of distributions, we draw 10⁶ samples from their sum and fit a negative binomial using moment matching. We compute the mean and standard deviation of the negative binomial distribution parameters across the bootstrap, and use this for our prior. The resulting priors are given in Table A.2.

Appendix B.1. Unseen data

An important way to validate a Bayesian model is by checking its predictions on unseen data, even if prediction is not the purpose of the model^10,18. If an NPI effectiveness model is entirely unable to extrapolate to unseen countries, we have strong reason to doubt its effectiveness estimates. However, we do not expect NPI effectiveness models to extrapolate perfectly. Almost always, unobserved factors such as changes in the ascertainment rate or IFR, spontaneous behaviour changes, or unobserved NPIs, will affect the observed number of cases and deaths. Our models ought to treat these factors as noise and not attribute their effects on R_t to the observed NPIs.

We use Leave-One-Out Cross-Validation (LOO-CV), fitting the model on 40 countries and extrapolating to the excluded country. We repeat this process for all 41 countries. In the excluded country, the first 14 days of case and death data are observed to estimate R_0,c as well as the initial outbreak sizes, and . All later days are unobserved (masked). The model uses the effectiveness estimates inferred from the 40 included countries and NPI activation dates to predict the course of the pandemic in the excluded country.

Figure B.7 visually shows extrapolations obtained from LOO-CV for a randomly chosen subset of six countries (all countries are shown in Figures C.18 to C.21). The predictive accuracy of the model, as expected, degrades with an increasing prediction horizon, suggesting the presence of unobserved factors. However, the predictive credible intervals are wide, and increase over time as noise in the growth rate accumulates. Importantly, our model makes calibrated forecasts in excluded countries (Fig. B.8), over periods of up to 2 months.

These are challenging predictions; to the best of our knowledge, no related study analyses how their estimated NPI effects generalise to unseen countries. Most related studies do not validate predictions on unseen data at all^19–23, reviewed in⁹. Flaxman et al.¹ hold out the last 14 days, from 20 April to 4th of May, for all countries in aggregate. However, the countries they study implemented all NPIs in March or earlier (Supplementary Table 2 in¹), meaning that in fact all countries were used to estimate NPI effects.

Note that Fig. B.8 includes the 6 countries that were used to select the hyperparameter (Appendix A). However, the calibration is similar if these 6 countries are excluded (Figure C.23).

Appendix B.2. Sensitivity Analysis

Sensitivity analysis reveals the extent to which results depend on uncertain parameters and modelling choices, and can diagnose model misspecification and excessive collinearity in the data²⁴. We vary many of the components of our model and recompute the NPI effectiveness estimates, summarised here. Further analysis can be found in Appendix C.

• Download figure
• Open in new tab

Figure B.7:

Model extrapolations on a randomly chosen subset of 6 excluded countries, produced using leave-one-out cross-validation. In each excluded country, the first 14 days of death and case data (solid dots) are observed; the model then extrapolates to all future days within the period of analysis (empty dots). For each country, we show the full window of analysis (from the start of the epidemic until the first NPI was lifted, or 30th of May 2020, whichever was earlier; see Methods). The shaded areas represent the 95% credible intervals.

For each sensitivity analysis, we quantify sensitivity using an easily interpretable measure shown above the legend of each plot: the average standard deviation, denoted . First, for each NPI, we compute the standard deviation of its median effect estimates across all experimental conditions in the given sensitivity analysis. We then average these across the NPIs. Since effectiveness is expressed in percentage reductions of R_t, the unit of is percent. Zero percent corresponds to no sensitivity.

We perform a multivariate sensitivity analysis on the key epidemiological priors. For computational reasons, all other sensitivity analyses are univariate.

Multivariate sensitivity to main epidemiological priors

The key epidemiological parameters in our model describe the delay from infection to case-confirmation, the delay from infection to death, and the generation interval. Our model places prior distributions over the means of these delay distributions. Since the priors already account for uncertainty in the delays, this analysis considers what would happen if our prior beliefs changed. Since the epidemiological parameters may interact in unexpected ways, we perform a multivariate sensitivity analysis, jointly changing the mean of the prior over the mean of each delay distribution (referred to as the “prior mean of the mean” below). We vary the prior means of the mean delays across a wide but epidemiologically plausible range of 5 values, resulting in 125 different experimental conditions. We present these results in two ways.

• Download figure
• Open in new tab

Figure B.8:

Calibration in held-out countries with leave-one-out cross validation. In each excluded country, the first 14 days of death and case data are observed to estimate R_0,c as well as the initial outbreak sizes; the model then extrapolates to all future days within the period of analysis. The plot shows the percentage of observed daily case and death counts that lie within the X% credible interval, across all countries and days. The identity line represents ideal calibration.

First, Figure B.9 shows the global sensitivity of our results to the prior mean of the mean of each distribution individually. For example, for each setting of the prior mean of the mean generation interval, we show the average over the 5 × 5 = 25 runs with different prior means placed over the mean delays between infection and case confirmation/death. This is equivalent to placing a uniform prior across the 125 experiment conditions and marginalising, and is standard methodology for computing global sensitivity to an individual parameter.

The prior means seem to mostly influence the relative effectiveness of business closures and gathering bans. Increasing the prior means of the infection-to-case-confirmation/death mean delays results in larger effectiveness estimates for gatherings bans and smaller estimates for business closures, while increasing the prior mean of the generation interval has the opposite effect.

Second, we assess the sensitivity of our results to jointly varying the prior means. This yields . We present this result graphically in Appendix C.1.

• Download figure
• Open in new tab

Figure B.9:

Sensitivity to key epidemiological parameter choices, evaluated using a multivariate analysis. We jointly vary the prior means over the mean of the generation interval, the delay between infection and case confirmation, and the delay between infection and death. Each panel displays sensitivity to the prior mean of one distribution, and the NPI effects are computed by averaging over the 25 runs with different prior means of the two other distributions (see text).

Sensitivity to country exclusions

Figure B.10 shows results if one country at a time is excluded from the data. As there is no strong justification for including or excluding any particular country, results ought to be stable if a country is excluded.

• Download figure
• Open in new tab

Figure B.10:

Sensitivity to excluding individual countries.

Sensitivity to case-undercounting

Figure B.11 shows results when scaling the recorded number of cases in each country. First, we correct the number of reported cases on each day using estimates of the time-varying ascertainment rate from Golding et al.²⁵. For example, if the estimated ascertainment rate in country c at time t is 50%, we double the corresponding number of reported cases. With this altered case data, the model estimates a larger effect for closing schools and universities, and a smaller effect for limiting gatherings to 1000 people or less. There is little change in the effects estimated for the other NPIs.

Second, we randomly either double or halve the reported number of cases in each country. This is intended to demonstrate that the model is invariant to constant differences in ascertainment between countries. As expected, there are negligible differences compared to the default results.

• Download figure
• Open in new tab

Figure B.11:

Sensitivity to case undercounting, considering both constant undercounting and time-varying undercounting.

Sensitivity to structurally different models

Figure B.12 shows the NPI effectiveness estimates from models that use only cases or deaths as observations, in contrast to our main model, which uses both. As expected, there are some differences in the NPI effect estimates between the models, which may be caused by the unique biases present in case data and in death data. Differences could also occur if NPIs differentially affected cases and deaths (e.g. if they affect different age groups). Nevertheless, the study’s qualitative conclusions, as outlined in the Discussion,^e hold across the different data types.

A number of implicit structural assumptions are made by the choice of model structure. We test sensitivity to these assumptions by evaluating NPI effectiveness estimates from alternative models, reproducing the structural sensitivity analysis from our concurrent work, where these models are described and compared in detail⁹. While these alternative model structures are also plausible, we chose the model described in the main text as it is relatively simple whilst also being highly robust⁹.

• Download figure
• Open in new tab

Figure B.12:

Sensitivity to different data types i.e., models using only confirmed cases, only deaths, or both.

The models are:

1. Different Effects Model. The effectiveness of each NPI is allowed to vary across countries.

2. Discrete Renewal Model. Instead of converting R_t into a daily growth rate, a renewal process is used as the infection model, as in earlier work^1,8,26–28. For computational reasons, we do not place a prior distribution over the generation interval parameters for this model.

3. Noisy-R Model. The noise terms affect R_t rather than the growth rate, as in Fraser⁸.

4. Additive Effects Model Each NPI has an additive effect on R_t. The joint effectiveness of a set of NPIs is produced by summing, rather than multiplying, their individual effectiveness estimates.

As Figure B.13 (left) shows, the multiplicative effect models support almost all conclusions drawn in the Discussion.^e The only exception is that the stay-at-home order NPI is categorised as “moderately effective” under the Different Effects Model (median reduction in R_t of 18%).

The results of the Additive Effects Model cannot be directly compared to the other models since it expresses results on a different scale (percentage reduction in R₀ instead of R_t). It is therefore shown in a separate panel. However, the trend in the results is visually similar to the default results.

Appendix B.3. Robustness to unobserved effects

Our data neither captures all NPIs which were implemented nor directly measures broader behavioural changes. Since these factors influence R_t, we must be wary of their effect being attributed to observed NPIs. We investigate this further by assessing how much effectiveness estimates change when previously unobserved factors are included and also when observed factors are excluded. This is best practice for addressing unobserved factors, such as confounders^29,30.

• Download figure
• Open in new tab

Figure B.13:

Structural sensitivity analysis. NPI effectiveness estimates under different structural assumptions. Left. Effectiveness estimates for multiplicative effect models. Note that, for computational reasons, the discrete renewal model does not have a prior over the generation interval. Right Effectiveness estimates for an additive effect models. For this model, the effectiveness of each NPI is expressed as a reduction of R₀ rather than R_t.

Unobserved factors can bias results if their timing is correlated with the timing of the observed NPIs³¹. The timings of our observed NPIs’ implementation dates are indeed mutually correlated, prompting the question of how much results change when we make previously observed NPIs unobserved. Figure B.14 (left) shows NPI effectiveness estimates when previously observed NPIs are excluded in turn. The study’s main qualitative conclusions^e hold across all experimental conditions, and the variations in NPI effectiveness are rarely large enough to cause a change in effectiveness category (Figure 5) except for the Gatherings limited to 1000 people or less NPI. Considering that some of the excluded NPIs have strong estimated effects when included, and are correlated with other NPIs, this degree of robustness is encouragingly high. It suggests that unobserved factors will not significantly bias results as long as their effects and their correlations with the studied NPIs do not exceed those of the studied NPIs. We hypothesise that this robustness to unobserved factors is due to the noise on the daily growth rate in our model⁹.

In addition, Figure B.14 (right) shows NPI effectiveness estimates when we observe (i.e. control for) additional NPIs, taken from the OxCGRT dataset³².

• Download figure
• Open in new tab

Figure B.14:

Robustness to unobserved effects. Left: Results when excluding previously observed NPIs. We exclude one of the NPIs in turn and show the estimates for the other NPIs. Note that this subplot shows the marginal effect for hierarchical NPIs rather than the total effect, different to other figures that show cumulative effects for gathering bans and businesses closures. For example, Figure 3 displays the total effect of closing most nonessential businesses, while here we show the additional effect of closing most nonessential businesses over just closing some high-risk businesses. We show the additional effects here because the effect of a cumulative intervention would become undefined when part of it is excluded from the analysis. Right: Results when controlling for previously unobserved NPIs. We include one additional NPI in turn and show the estimates for the NPIs in our study (the additional NPI is not shown).

Appendix C.1. Multivariate Sensitivity Analysis - Expanded Results

We now present the results of our joint multivariate sensitivity analysis (previously summarised in Figure B.9), in which we vary the mean of the prior (“prior mean”) over the mean of the generation interval, the delay between infection and death, and the delay between infection and case confirmation.

Figure C.15 shows, for each NPI, how NPI effects change when all prior means are varied simultaneously.

We find that the most sensitive NPIs are Gatherings limited to 1000 people or fewer, Gatherings limited to 100 people or fewer and Most nonessential businesses closed. However, the largest differences appear when all of the parameters are set to the most extreme values that jointly produce a change in a particular direction. We also see systematic trends as the parameters are varied, e.g., increasing the prior mean of the generation interval mean tends to decrease the effectiveness of the Gatherings limited to 1000 or fewer NPI, while increasing prior means of the infection to death/case confirmation distribution means tends to increase the effectiveness of this NPI.

• Download figure
• Open in new tab

Figure C.15:

Global sensitivity analysis. Each cell shows the difference between the median effectiveness estimate under a specific experimental condition and the default condition, where the estimates are expressed as percentage reduction in R_t. Each subplot represents a particular prior mean of the mean generation interval and an NPI. The NPIs vary top to bottom and the generation interval prior mean increases left to right. Each cell with a subplot represents a particular choice of the prior mean of the mean infection to death delay (increasing left to right) and the prior mean of the mean infection to case confirmation delay (increasing bottom to top). Positive cell values indicate that the median NPI estimate is larger than with default settings and vice versa.

Appendix C.2. Sensitivity to additional epidemiological assumptions

For completeness, Figure C.16 shows sensitivity to two further epidemiological assumptions. On the left, we show the dependence of NPI effectiveness estimates on R₀. We place a Normal prior over R₀, and a hyperprior over its variance, reflecting the wide disagreement of regional estimates of R₀. In the sensitivity analysis, we vary the mean of the prior from a low value (2.38) to a high one (4.28). As expected, higher mean R₀ values result in larger NPI effects. This trend is apparent in all NPIs but stronger for NPIs that tended to be implemented early on (like school closures and gathering bans).

On the right, we vary the prior over NPI effectiveness. Our default prior on NPI effectiveness is assymmetric, reflecting a belief that NPIs are more likely to reduce R_t than increase it. Here, we additionally test a symmetric Normal prior, and a one-sided Half-Normal prior, which does not allow for NPIs to increase R_t.

Appendix C.3. Sensitivity to data preprocessing

When the case count is small, a large fraction of cases may be imported from other countries and the testing regime may change rapidly. To prevent this from biasing our model, we neglect case numbers before a country has reached 100 confirmed cases. Similarly, we neglect death numbers before a country has reached 10 deaths. Here, we vary these thresholds. Intuitively, the minimum cases threshold should be higher than the minimum deaths threshold.

• Download figure
• Open in new tab

Figure C.16:

Sensitivity to additional epidemiological priors. Left: Sensitivity to the prior mean on R₀. Right: Sensitivity to the NPI effectiveness prior.

• Download figure
• Open in new tab

Figure C.17:

Data preprocessing sensitivity. Left. Variations in the minimum number of cases. Right. Variations in the minimum number of deaths.

Appendix C.4. Validation by predicting unseen data - all countries

Here we show the country-level plots of the single-country holdout experiments from Ap- pendix B.1. We use leave-one-out cross-validation: fitting the model on 40 countries and showing its predictions on the excluded country. We repeat this process for all 41 countries. In the excluded country, the first 14 days (filled dots) of death and case data are observed to allow roughly inferring R₀. These days are not used to infer NPI effectiveness. All later days are masked (unfilled dots). The activation dates of NPIs are also given, and the model uses the effectiveness estimates inferred from the 40 other countries.

• Download figure
• Open in new tab

Figure C.18:

Predictions on excluded countries.

• Download figure
• Open in new tab

Figure C.19:

Predictions on excluded countries.

• Download figure
• Open in new tab

Figure C.20:

Predictions on excluded countries.

• Download figure
• Open in new tab

Figure C.21:

Predictions on excluded countries. Vertical lines show the activation (or inactivation) dates of NPIs. Shaded areas are 95% credible intervals. Blue and red dots show the observed confirmed cases and deaths, while blue and red lines show the median model estimates of cases (C_t) and deaths (D_t). Empty dots are not observed by the model. For each country, we show the full window of analysis (from the start of the epidemic until the first NPI was lifted, or the 30th of May 2020, whichever was earlier; see Methods).

Appendix C.5. Posterior predictive distributions

The posterior predictive distribution (Figure C.22) shows the predicted number of cases and deaths after observing the data. Although these curves can be called ‘fits’, the degree of fit to the data must be interpreted with great care. The fit is generally tight, but this is partly due to the inferred latent noise variables and . Inferring this latent noise allows the posterior predictive distribution to closely match the data without overfitting the effectiveness parameters to the data. Such behavior is common in Bayesian models, which often perfectly interpolate the data without overfitting³³. The noise terms can account for periods where infections grew faster or slower than predicted based solely on the active NPIs. In such periods, the noise may account for changes in testing, reporting, and unobserved interventions.

• Download figure
• Open in new tab

Figure C.22:

Left: Posterior predictive distributions for two exemplary countries. See text. Right: Inferred R_t over time.

Appendix C.6. Calibration without countries used for hyperparameter selection

• Download figure
• Open in new tab

Figure C.23:

Calibration in held-out countries with leave-one-out cross-validation. Here we include only those 35 countries that were not used to select the hyperparameter. We show the first 14 days of cases and deaths in each country and then extrapolate to future days within the period of analysis. The plot shows the percentage of observed daily case and death counts that lie within the X% credible interval, across all countries and days.

Appendix C.7. MCMC stability results

• Download figure
• Open in new tab

Figure C.24:

MCMC stability results. Left: R-hat statistic. Values are close to 1, indicating convergence. Right: Relative effective sample size. A value of 1 indicates perfect decorrelation between samples. Values above (below) 1 indicate that the effective number of samples is higher (lower) than the actual number of samples due to negative (positive) correlation, respectively.

Appendix D.1. Estimated R₀ by country

Appendix D.2. Collinearity

Appendix D.2.1. The individual effects of school and university closures

The dates of school and university closures coincide nearly perfectly for every country except Iceland and Sweden, which closed universities but not schools (Figure 1). As a consequence, the inferred individual effects depend strongly on the inclusion or exclusion of these countries in the dataset (Figure D.25). If we included all countries, we would conclude that university closures were more effective than school closures (black markers in Figure D.25). However, if we excluded Iceland or Sweden, we would conclude that they were roughly equally effective. As there is no strong justification for including or excluding one particular country, we cannot meaningfully disentangle the effects of school and university closures. However, there is much more data to determine the joint effect, and it is indeed much more stable (Figure D.25).

• Download figure
• Open in new tab

Figure D.25:

The individual effectiveness of closing schools and of closing universities, as well as the joint effect of closings school and universities, estimated on all countries (default), all countries except Sweden, and all countries except Iceland. Median, 50% and 95% credible intervals are shown.

Appendix D.2.3. Correlations between effectiveness estimates

The effectiveness parameters α_i are typically negatively correlated with each other for NPIs which are often used together, reflecting uncertainty about which NPI is reducing R. Excessive collinearity in the data would result in wide posterior credible intervals with strong correlations²⁴, but we find weak posterior correlations between effectiveness estimates. The strongest correlation between any pair of NPIs is − 0.42, between “closing schools and universities” and “closing some businesses” (Figure D.26). The weak correlations are one indicator that collinearity is manageable with our dataset.

Effect on NPI combinations

To better understand posterior correlations, we visualize their effect in hosted video files. As we condition on different values for one NPI, we can see that the estimates of other NPIs change only slightly, always staying well within the credible intervals in Figure 3. The significance of posterior correlations is small enough that it is possible to calculate a reasonable approximation to the mean effect of a set of NPIs by simply combining the mean percentage reductions for each individual NPI (e.g. two 50% reductions lead to a 75% reduction). For example, this approximation leads to a joint effect of 75% for all NPIs together, which closely matches the exact mean joint effect of 77%.

Videos are available online here.

• Download figure
• Open in new tab

Figure D.26:

Posterior correlations between effectiveness parameters α_i.

Appendix D.3. Posterior Epidemiological Parameter Distributions

Figure D.27 shows the posterior distributions for variables describing the key delay distributions in our model: the generation interval, the delay between infection and case confirmation, and the delay between infection and death. We find that the posterior distributions of these parameters are somewhat tighter than their priors, but they still explore a wide range of values. This suggests that the data provides evidence, albeit weak, about these delay distributions (for example, from visual inspection, the data would show that the delay is longer for deaths than for cases). An exception is the dispersion of the infection to death delay distribution, which has a very wide prior.

• Download figure
• Open in new tab

Figure D.27:

Posterior distributions over key epidemiological parameters describing the generation interval and the delays between infection and case confirmation/death.

Appendix E.1. Limitations of the data

We only record NPIs if they are implemented in most of a country (if they affect more than three-quarters of the population). We thus miss NPIs which were only implemented regionally. For example, a few regions in Germany implemented stay-at-home orders, but most did not. Thus, Germany is listed as “no stay-at-home order” in our data. Additionally, our NPI definitions were not perfectly granular. For example, a gathering ban on gatherings of >15 people and a ban on gatherings of >60 people would both fall under the NPI “Gatherings limited to 100 people or less”, despite likely having different effects on R_t. Finally, while we included more NPIs than previous work (Table F.7), there are many NPIs for which we were not able to collect enough high-quality data for our modeling, such as public cleaning or changes to public transportation.

Of the 41 countries in our dataset, 33 are in Europe. As a result, the NPI effectiveness estimates may be biased towards effects in Europe, and NPI effectiveness may have been different in other parts of the world.

Appendix E.2. Model limitations

Situation: Sometimes only public gatherings are banned, with no explicit ban on private gatherings

How we deal with it: We still counted this as a ban on gatherings.

Examples:

• Sweden: In Sweden, they banned all public gatherings of more than 50 people (demonstrations, religious meetings, theater performances, markets, and other events that relied on the constitutional freedom of assembly), however, the ban did not have a mandate to prohibit private gatherings (such as private parties). We counted this as a ban on gatherings.

• Finland: In Finland, they banned all public gatherings of more than 10 people on the 16th of March. Although formal restrictions did not apply to private gatherings, this policy met our definition of a ban on gatherings. (Note that this inclusion seems particularly valid in light of the fact that, according to Finnish police, the formal restrictions on public events were widely interpreted to apply to private gatherings as well, and there were very few reports of large private parties despite the absence of formal restrictions.)

Situation: The size limits on gatherings sometimes differ between indoor and outdoor gatherings

How we deal with it: In these cases, we relied on the limitations on indoor events, as these events entail a greater risk of transmission.

Example:

• Spain: In Spain, a range of rules were employed as the country gradually eased restrictions on gatherings. In phase 1, cultural events were permitted with up to 30 people indoors and up to 200 outdoors. This was counted as “Gatherings limited to 100 people or less.”

Situation: The size limit on gatherings sometimes differs between different types of gatherings

How we deal with it: In this case, researchers would use their best judgment to infer whether the restriction would apply to most gatherings of a given size.

Example:

• Spain: In Spain, phase 1 of the reopening allowed for cultural events to have up to 30 participants indoors, while social gatherings were limited to 10 people. In this case, since “cultural events” is broad, we counted this as a case of “gatherings limited to 100 people or less.” However, if for example all gatherings above 5 people had been banned with an exception for funerals, we would have counted this as “gatherings limited to 10 people or less,” since the exemption only applied to a minority of gatherings.

Situation: Limitations on gathering sizes are not clearly given, yet a policy stating that “large events are banned” is in place

How we deal with it: Our researchers used the relevant context to infer the most likely scope of the policy.

Example:

• Albania: on March 8 “authorities had also ordered cancellations of all large public gatherings including cultural events and were asking sporting federations to cancel scheduled matches”. The events that are mentioned here are multi-thousand person gatherings, and so we took March 8th to be the start date of “Gatherings limited to 1000 people or less”. However it was unclear whether gatherings of 100-1000 would also have been banned, so we did not yet say that “Gatherings limited to 100 people or less” was instantiated.

Situation: Only some schools were closed, or schools reopened gradually

How we deal with it: Since our definition of the NPI is that “Most schools are closed”, we did not count the closure of just a few schools or school years sufficient to meet this criteria. Similarly, if schools reopened for only a very limited number of year groups, for example for final year students sitting exams, we did not count this as a lifting of the “most schools closed” NPI.

Examples:

• Sweden: Sweden kept all schools through 9th grade open, but closed high schools (>16 year olds). In this case, we did not count this as “Most schools closed”, since more than 75% of students are below 9th grade.

• Czech Republic: After closing all schools on March 13, the Czech Republic allowed schools to reopen for teaching in some contexts from May 11 (specifically for students in their final year of primary school or high school preparing for exams). However, we still counted this as “Most schools closed” since the majority of students were not in school. We recorded the end date for school closure to be June 8, when all schools reopened.

Situation: In a country where most non-essential businesses were closed, the lifting of business closures is gradual, and businesses in different sectors are successively allowed to open

How we deal with it: Countries reopen sectors in different, idiosyncratic ways and successions. Given the available data, it is not feasible to create a principle that can be applied unambiguously to every single case without some involvement of researcher judgment. The general guideline we used was: If only a few, low-risk businesses (e.g., bike stores, hardware stores, etc.) are additionally allowed to reopen, then we still counted this as “Most nonessential businesses closed.” However if any one of the following criteria are met, then we counted “Most nonessential businesses closed” as having lifted, but the “Some businesses closed” NPI was still in place:

• All regular retail stores, with only a few exceptions e.g. size limitations, are open

• Contact-based services, such as hairdressers and tattoo parlors, are open

• Restaurants and bars are open and serving indoors

We decided that meeting any one of these criteria is a sufficient condition for taking a country from “Most nonessential businesses closed” to “Some businesses closed.” This heuristic was partly based on the fact that the status of these categories appeared to be consistently correlated, meaning that, even in the absence of complete specifications as to what had reopened or not, it was typically possible to infer the overall level of reopening based on either of these categories. Meeting at least one of these criteria was considered a necessary condition for ending the “Most nonessential businesses closed” NPI.

Examples:

• Slovakia: On April 22, retail operations and services up to 300 m² opened. Since this meets one of the sufficient conditions, we counted April 22 as the end date for “Most nonessential businesses closed”

• Ireland: On May 18, the following reopened: hardware stores, builders, merchants and those providing essential supplies, retailers involved in the sale and repair of vehicles, certain office supply stores. Because this white list does not meet any of the three criteria, Ireland’s end date for “Most nonessential businesses closed” was not counted as May 18.

• Czech Republic: On April 20, several businesses reopened, including farmer’s markets, marketplaces, locksmiths, bike shops, car dealers, electronics stores. At this point, none of the criteria were met, so we recorded the Czech Republic as still having “Most nonessential businesses closed”. On May 11, a long list of businesses reopened, including barbers, hairdressers, museums, all establishments in sufficiently large shopping centers, shows with up to 100 participants, and restaurants with a window facing the street. Since contact-based services (hairdressers) and all retail establishments in sufficiently large spaces were allowed to reopen, we counted May 11 as the end date for the “Most nonessential businesses closed” NPI.

• Croatia: On April 27, all “trade activities” (except within shopping malls), service jobs that don’t involve physical contact, museums, libraries, and galleries opened. Since the criteria regarding “all retail stores being open” was met, we counted April 27 as the end date for “Most nonessential businesses closed”.

Data

1. NPI Activations: ϕ_i,t,c ∈ {0, 1}.

2. Observed (Daily) Cases: Ct,c.

3. Observed (Daily) Deaths: D_t,c.

Prior Distributions

1. Country-specific R₀: R_0,c ∼ Normal(3.25, κ); κ ∼ Half Normal(µ = 0, σ = 0.5).

2. NPI effectiveness: α_i ∼ Asymmetric Laplace(m = 0, κ = 0.5, λ = 10). m is the location parameter, κ >0 is the asymmetry parameter, and λ >0 is the scale parameter.

3. Infection Initial Counts:

4. Observation Noise Dispersion Parameters:

Hyperparameters

1. Growth Noise Scale, σ_g = 0.2.

Delay Distributions

1. Generation interval distribution^13,14:

2. Time from infection to case confirmation 𝒯 ^(C)13,15,16:^f

   This distribution is converted into a forward-delay vector: i.e., the delay follows a truncated and normalised negative binomial distribution.

3. Time from infection to death 𝒯 ^(D)f13,15,17:

   This distribution is converted into a forward-delay vector: i.e., the delay follows a truncated and normalised negative binomial distribution.

Infection Model

Observation Model^f