The limits of estimating COVID-19 intervention effects using Bayesian models ============================================================================ * Patrick Bryant * Arne Elofsson ## Abstract To limit the rapid spread of COVID-19, most governments have introduced different non-pharmaceutical interventions, which might have severe costs for society. Therefore, it is crucial to evaluate the most cost-effective interventions, using, for instance, Bayesian modelling. Such modelling efforts have deemed lockdown to account for 81% of the reduction in R​, contributing to government policies. Here, we show that these conclusions are unsupported and that policies therefore should not be based on these studies. *[https://www.eurosurveillance.org/for-authors](https://www.eurosurveillance.org/for-authors)* ## Background Due to the rapid spread of COVID-19 across Europe, and the quick increase in cases and deaths in some countries, European countries introduced non-pharmaceutical interventions to limit virus spread1. Efforts to elucidate the impact of interventions through computational modelling have been made2–4. Still the actual effects of these interventions are hard to differentiate from each other, as many interventions were introduced almost simultaneously. This communication focuses mainly on the deficiencies with a recent Bayesian model used to estimate the effect of five different interventions across 11 European countries2. Still, it is also relevant for other modelling attempts, including our own, where we try to use mobility in various sectors of society to estimate their effects5. A recent study2 uses MCMC simulations6,7 to infer impacts on the basic reproductive number (R​) through governmental non-pharmaceutical interventions (NPIs). From the estimated R​, the number of daily cases is predicted, which then through an infection-to-death distribution, is used to infer the number of daily deaths. The model fits the number of daily deaths to the observed number of deaths8 using a Bayesian posterior distribution. The paper reports that current interventions have been sufficient to drive R​ below 1 (which is necessary for the number of infected people to decrease) for all 11 European considered countries. They *“show that major non-pharmaceutical interventions and lockdown, in particular, have had a large effect on reducing transmission”*. The authors conclude that lockdown accounts for 81% [75% – 87%] of the reduction in R​, i.e. that all other interventions only have minimal effects on the decline of R​. ## The issue The peculiar aspect of the claim that lockdown accounts for 81% of the reduction in R​, is that Sweden did not implement any lockdown, but still see a similar decrease in R​ as the other countries, even though the other NPIs were reported to have no substantial effect on R​. To solve this problem, as compared with the authors’ earlier work9, which showed a significantly higher R​ for Sweden, they invoke a country-specific last intervention parameter, which is only implemented for Sweden10 (see equation i). The “last intervention” parameter is multiplied with R​, and can therefore be seen as a parameter adjusting the model for Sweden independently. As can be seen, when analysing the posterior distributions of the intervention parameters, the “last intervention” parameter for Sweden results in 73.5 % of Sweden’s reduction in R​ (Figure 1). The last intervention impact on R​ is not reported or discussed in the Nature publication, possibly misleading decision-makers on the importance of lockdowns. ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/08/16/2020.08.14.20175240/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2020/08/16/2020.08.14.20175240/F1) Figure 1. Posterior distributions of relative reduction in R​ for each intervention and the “last intervention” parameter only introduced for Sweden. The median reductions are –0.7%, –0.7%, –0.7%, 81.1% and –0.6 % for NPIs Schools and Universities, Self-isolating if ill, Public events, Lockdown and Social distancing encouraged respectively. The last intervention, only implemented for Sweden, has a median reduction of 73.5%. This is the equation used for estimating the impact of NPIs on R​ in the MCMC simulations ![Formula][1] where I​m, 1–7 states if country m has introduced the following interventions, on the given day, (1) closing of schools and universities, (2) Self-isolating if ill, (3) banning of public events, (4) the first intervention introduced, (5) lockdown, (6) social distancing encouraged and (7) last intervention. The additional country-specific parameters added for the lockdown (here referred to as “extra lockdown”) and last intervention are sampled from the following distributions: lockdown ∼ normal(0,gamma) last intervention∼ normal(0,gamma) gamma ∼ normal(0,.2) To analyze the importance of the last intervention option only introduced for Sweden, we remove it (see equation ii). The resulting reduction in R​ for Sweden is identical within statistical margins, but the fit to the daily deaths is worse (Figure 2). Interestingly, the banning of Public events (Sweden’s last introduced NPI) now becomes more important (median reduction of 50.1%), being almost on par with that of the lockdown (median reduction of 64.9%, Figure 3). ![Formula][2] ![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/08/16/2020.08.14.20175240/F2.medium.gif) [Figure 2.](http://medrxiv.org/content/early/2020/08/16/2020.08.14.20175240/F2) Figure 2. Modelling results for Sweden with (A, B, C) and without (D, E, F) the introduction of the “last intervention” parameter. The thick blue lines correspond to the means, while the darker and lighter shades of blue correspond to the 50 and 95 % confidence intervals respectively. ![Figure 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/08/16/2020.08.14.20175240/F3.medium.gif) [Figure 3.](http://medrxiv.org/content/early/2020/08/16/2020.08.14.20175240/F3) Figure 3. Posterior distributions of relative % reduction in R​ if the last intervention option only introduced for Sweden is removed. The median reductions are –0.7%, –0.7%, 50.1%, 64.9% and –0.1 % for NPIs Schools and Universities, Self-isolating if ill, Public events, Lockdown and Social distancing encouraged respectively. Finally, suppose the “extra lockdown” introduced for all countries except for Sweden and the “last intervention” options only added for Sweden are removed (equation iii). In that case, the banning of public events now becomes the NPI with the most importance (reduction of 78%). Now also the effect of the closing of schools and universities, and social distancing rise (reductions of 5.4%, and 5.3% respectively), while the self-isolating if ill importance remains unchanged (median 0.7 %), and the lockdown importance greatly diminishes (0.2 %), Figure 4. ![Formula][3] ![Figure 4.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/08/16/2020.08.14.20175240/F4.medium.gif) [Figure 4.](http://medrxiv.org/content/early/2020/08/16/2020.08.14.20175240/F4) Figure 4. Posterior distributions of relative % reduction in R​ if the extra lockdown option and the last intervention option only introduced for Sweden is removed (see equation iii). The median decreases are 5.4%, –0.7%, 77.9&, 0.2% and 5.3 % for NPIs Schools and Universities, Self-isolating if ill, Public events, Lockdown and Social distancing encouraged respectively. ## Discussion and Conclusion The Bayesian model used by ICL to estimate the impact of each intervention appears to be sensitive to small perturbations. The reason the lockdown seems to have such high importance is likely due to it being the last intervention implemented, and not because it had the most substantial impact. We highlight this by showing that the importance of the “last intervention” diminishes when treating Sweden in the same way as all other countries. Further, the modification of the initial model9 enables adjustments to the daily deaths from the model in each country. It thus creates the possibility of fitting a wide range of data to the observations but loses the ability to identify cross-country patterns. We believe that allowing country-specific effects of the NPIs makes it very difficult to estimate the relative impacts of NPIs, since there is no possibility of distinguishing between NPI effects and other country-specific effects, e.g. population density. In conclusion, it is peculiar that the model displays an almost identical change in R​ in all countries, dropping sharply below one at the final NPI, independently on the nature of that NPI. In reality, all countries had different NPIs implemented at different time points, likely with varying strength and efficiency, and it is quite likely that NPIs such as enforcing social distancing at least had some effects, not seen in the models. Given the importance the initial report had on government policies and the fact that we show here that the conclusions made about the significance of the lockdown are not entirely correct, we do think that we should pinpoint this to readers and policymakers. Correct assumptions on the effects of NPIs are becoming even more urgent as many nations still are imposing different NPIs, and that these might go on for an extended period ([https://www.bbc.com/news/explainers-53640249](https://www.bbc.com/news/explainers-53640249), last accessed 20200811). ## Data Availability All data and code is available from [https://github.com/patrickbryant1/COVID19.github.io/tree/master/ICL\_critique](https://github.com/patrickbryant1/COVID19.github.io/tree/master/ICL_critique) [https://github.com/patrickbryant1/COVID19.github.io/tree/master/ICL\_critique](https://github.com/patrickbryant1/COVID19.github.io/tree/master/ICL_critique) ## Declaration of interests We declare no competing interests. ## Funding statement ### Financial support Swedish Research Council for Natural Science, grant No. VR-2016–06301 and Swedish E-science Research Center. Computational resources: Swedish National Infrastructure for Computing, grant No. SNIC-2019/3–319. ## Acknowledgements We acknowledge Claudio Bassot’s contribution by sharing the Imperial College. Without this information, this study would not be possible. We are also grateful to various colleagues and friends that contributed to the discussion. Finally, we thank the authors of the Imperial College London report for making their data and model freely available. * Received August 14, 2020. * Revision received August 14, 2020. * Accepted August 15, 2020. * © 2020, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. 1.Team, E. E. & Eurosurveillance editorial team. Updated rapid risk assessment from ECDC on coronavirus disease 2019 (COVID-19) pandemic: increased transmission in the EU/EEA and the UK. Eurosurveillance vol. 25 (2020). 2. 2.Flaxman, S. et al. Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe. Nature (2020) doi:10.1038/s41586-020-2405-7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41586-020-2405-7&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32512579&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F16%2F2020.08.14.20175240.atom) 3. 3.Giordano, G. et al. Modelling the COVID-19 epidemic and implementation of population-wide interventions in Italy. Nat. Med. 26, 855–860 (2020). [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41591-020-0883-7&link_type=DOI) 4. 4.Dehning, J. et al. Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions. Science 369, (2020). 5. 5.Bryant, P. & Elofsson, A. Estimating the impact of mobility patterns on COVID-19 infection rates in 11 European countries. doi:10.1101/2020.04.13.20063644. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMC4wNC4xMy4yMDA2MzY0NHYyIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjAvMDgvMTYvMjAyMC4wOC4xNC4yMDE3NTI0MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 6. 6.Ugarte, M. D. Banerjee, S. Carlin, B. P., and Gelfand, A. E. Hierarchical Modeling and Analysis for Spatial Data. Second Edition. CRC Press/Chapman & Hall. Monographs on Statistics and Applied Probability 135, Boca Raton, Florida, 2015. 562 pp. ISBN-13: 978-1. Biometrics vol. 71 274–277 (2015). 7. 7.Website. Stan [Internet]. stan-dev.github.io. [cited 2020 Apr 23]. Available from: [https://mc-stan.org](https://mc-stan.org). 8. 8.Download today’s data on the geographic distribution of COVID-19 cases worldwide. European Centre for Disease Prevention and Control [https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-disribution-covid-19-cases-worldwidet](https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-disribution-covid-19-cases-worldwidet) (2020). 9. 9.Report 13 – Estimating the number of infections and the impact of non-pharmaceutical interventions on COVID-19 in 11 European countries. Imperial College London [http://www.imperial.ac.uk/medicine/departments/school-public-health/infectious-disease-epidemiology/mrc-global-infectious-disease-analysis/covid-19/report-13-europe-npi-impact/](http://www.imperial.ac.uk/medicine/departments/school-public-health/infectious-disease-epidemiology/mrc-global-infectious-disease-analysis/covid-19/report-13-europe-npi-impact/). 10. 10.ImperialCollegeLondon. ImperialCollegeLondon/covid19model. GitHub [https://github.com/ImperialCollegeLondon/covid19model](https://github.com/ImperialCollegeLondon/covid19model). [1]: /embed/graphic-2.gif [2]: /embed/graphic-3.gif [3]: /embed/graphic-6.gif