Abstract
Background In mid April 2020, with more than 2 · 5 billion people in the world following social distancing measures due to COVID-19, governments are considering relaxing lock-down. We combined individual clinical risk predictions with epidemic modelling to examine simulations of isolation and exit policies.
Methods We developed a method to include personalised risk predictions in epidemic models based on data science principles. We extended a standard susceptible-exposed-infected-removed (SEIR) model to account for predictions of severity, defined by the risk of an individual needing intensive care in case of infection. We studied example isolation policies using simulations with the risk-extended epidemic model, using COVID-19 data and estimates in France as of mid April 2020 (4 000 patients in ICU, around 7 250 total ICU beds occupied at the peak of the outbreak, 0·5% percent of patients requiring ICU upon infection). We considered scenarios varying in the discrimination performance of a risk prediction model, in the degree of social distancing, and in the severity rate upon infection. Confidence intervals were obtained using an Approximate Bayesian Computation approach. The framework may be used with other epidemic models, with other risk predictions, and for other epidemic outbreaks.
Findings Based on the data for France as of mid April 2020, simulations indicated that an exit policy considering clinical risk predictions starting on May 11, as planned by the government, could enable to immediately relax restrictions for an extra 10% (6 700 000 people) or more of the lowest-risk population, and consequently relax the restrictions on the remaining population up to two times (or several months) faster, with only a small proportion of the population remaining in isolation for an extended period of time – while abiding to the current ICU capacity. In contrast, implementing the same exit policy without risk predictions would exceed the ICU capacity by a multiple. Sensitivity analyses showed that when the assumed percentage of severe patients among the population decreased, or the prediction model discrimination improved, or ICU capacity increased, policies based on risk models had a greater impact on the results of epidemic simulations. At the same time, sensitivity analyses also showed that differential isolation policies require that higher risk individuals comply with recommended restrictions. In general, our simulations showed that risk prediction models could always improve policy effectiveness, keeping everything else constant, in line with value of information arguments, even for models with moderate discrimination power.
Interpretation Clinical risk prediction models should be considered to manage outbreaks using a framework as the one developed. They can inform personalised isolation policies, for example by gradually restricting (relaxing) isolation from the highest (lowest) to the lowest (highest) predicted risk individuals, when such policies are considered. This may lead to both safer and faster outcomes than what can be achieved without such prediction models. They enable personalisation of policies, which are known to improve effectiveness in other non-healthcare contexts.
Funding No funding was used for this research.
Evidence before this study Several countries have implemented non-pharmaceutical interventions based on social distancing and isolation measures in order to limit the spread of COVID-19. There has been limited differentiation in the degree of isolation measures, except for those critical for the functioning of the healthcare system and other key services. There is limited evidence about the impact of relaxing these measures as this has happened only recently and in only a few countries. Investigating the potential impact of differential restrictions depending on medical factors, such as the risk of severe symptoms if infected by Sars-Cov-2, may inform policies for imposing or relaxing isolation policies when these are considered.
Added value of this study This study investigates incorporating clinical risk predictions in epidemic models, allowing to explore isolation policies that consider individual clinical risks using simulations.
Implications of all the available evidence Epidemic simulations of isolation policies that consider predicted clinical risks in order to differentiate restrictions depending on risks indicate the feasibility of new, possibly gradual, policies that may be riskier to implement without undertaking this type of risk-based approach.
1. Introduction
As of April 2020, many countries have adopted non-pharmaceutical interventions, such as isolation restrictions,1 to control the spread of COVID-19. Epidemic models have been used to inform such policies.2,3,4 Governments now consider relaxing these restrictions. Immunity tests5 and technology6 may support such policy decisions. We considered an application of predictive technologies, such as machine learning, that can be used to better understand outbreaks using epidemic simulations: using personalised predictions of severity risk, defined as requiring ICU if infected, in epidemic simulations to examine differential risk based isolation policies. Other clinical risks can be considered similarly.
We extended standard epidemic models, namely a version of SEIR,7 to incorporate personalised risk predictions. Using simulations, we investigated how prediction models for patient severity may inform policy in two scenarios. First, when there is an ongoing outbreak as it was the case in France on the 17th of March, 2020, when lock-down started. Second, when the outbreak has been curbed by lock-down and progressive loosening of isolation policies (”exit”) may take place, as planned in France starting from the 11th of May, 2020.
There has already been research indicating differential impact of COVID-19 across patients, for example depending on hypertension, diabetes or other factors,8,9,10 but there is currently no standard risk prediction model considering all factors, although some early versions are available.11,12 Therefore we assumed hypothetical risk prediction models and studied the sensitivity of the simulation results with respect to model discrimination.
To populate the models, we used available COVID-19 estimates and data from France as of mid April 2020.13 At the time, there were about 4 000 beds occupied by people with COVID-19, with a peak at 7 148 people in intensive care, to be compared to the overall roughly 10 000 capacity recently reached by the French health system. We used current estimates with a reproduction number value of 
prior to lock-down, and an estimate of 2 · 5million people who had been infected (immune or infected at the time) when it started on March 17.14
Simulations indicated that, keeping other factors constant, using risk predictions can allow lower percentage of the population under stricter isolation restrictions, or equivalently differential relaxing of restrictions earlier or safer. The results were robust for a wide range of model discrimination assumptions, and stronger when the proportion of severe cases in the population was lower. Sensitivity analysis was also performed on 2 parameters capturing the degree of social distancing in the high and low risk populations, and on the availability of ICUs.
Related work
Since its outbreak in December 2019, a number of papers have attempted to forecast the COVID-19 spread15 using epidemic models. The most common approach uses compartmental models,16 i.e. assumes that the population is 65 divided into compartments, relative to their infection state, and that individuals belonging to the same compartment exhibit similar behaviour.
Among them, the susceptible-infected-removed (SIR) model is perhaps the most commonly adopted, mainly due to its simple computation and quick adaptation to new diseases, as it relies on solving a set of ordinary differential equations (ODEs). Each individual typically progresses from healthy (’S’ for susceptible to get infected), to infectious (’I’) and finally to the recovered state (’R’) also seen as a permanent immunity state (at least, at a reasonable time scale).
In the COVID-19 case, consistent SIR variants have been used as alternatives to the standard model, as they make for a better fit of the actual diffusion of the virus. In particular, the SEIR model includes, in addition to the standard states, an incubation state (’E’ for exposed/latent).17,18 Finer versions have been also explored, namely SEAIR (’A’ for asymptomatic infectious),19,1 SEIR-SD (’SD’ for social distancing),20,21 and many others,22,23,24 each of which implies a subdivision of the standard S, I and R compartments.
Accurate estimation of the model parameters and data collection are critical for the aforementioned models, especially as the number of compartments (and therefore of parameters) increases.25,26 Two other approaches, though less commonly used in the COVID-19 case, are agent-based,27,28,3 and network-based,29 the latter being intermediate in complexity between agent-based and compartmental models.
Numerous containment/lock-down strategies have been proposed to flatten the predicted curve of the number of severe cases by means of non-pharmaceutical interventions such as strict quarantine, 30 social distancing, strict hygiene, population screening, etc.4, 31, 19, 32 Although social distancing might be applied to each individual in a similar way, a different policy consists of adjusting it to the seriousness of the symptoms one has if infected.33,34,35 However, severity of symptoms upon infection is not known a priori and, at best, can only be predicted. Such predictions may be possible using predictive models, for example using data science and machine learning methods and principles, based on data related to infection symptoms.11,12
2. Methods
Risk-Extended SEAIR Model
The model introduced extends the standard SEAIR model in two directions. First, it considered an additional compartment, denoted by U, to account for people admitted in ICU. Second, it considered that any individual may fall into four different categories depending on the risk prediction made (high or low) and the isolation restrictions recommended (high or low). More precisely, each individual may be considered: (a) high risk and submitted to high isolation recommendations (true positive), (b) high risk and submitted to low isolation restrictions (false negative), (c) low risk and submitted to high isolation recommendations (false positive), (d) low risk and submitted to low isolation restrictions (true negative). As a whole, this led to 22 possible compartments.
We considered that the control parameter of a policy applying differentiated isolation restriction w.r.t. to the predicted risk of developing severe symptoms is the proportion ρ of individuals categorised in the low isolation restrictions group. We denote with p the fraction of individuals with symptoms not requiring ICU if infected. Depending on the accuracy of the prediction model, the rates of false positives and false negatives, denoted by qFP and qFN respectively, vary but obey the following equation, for given p and ρ:
We assumed that people change their behaviour depending on the restrictions introduced by the differentiated policy, modelled using two behavioural variables denoted by δr (for the group with low isolation restrictions/low predicted risk) and δc (for the group with high isolation restrictions/high predicted risk), in [0,1], with δr < δc. Variables δr and δc capture a level of protection, which may aggregate several factors such as respiratory and hand hygiene, how much a person has lowered the number of exits from home and the social interactions, etc. How individuals of a given category reduce their contacts not only depends on δc and δr but also on the proportion ρ of people with less strict restrictions. The decrease in contact rates, cr and cc for the two groups, may be obtained with the following formula:
These parameters have been used in the literature modelling the current lock-down.36,13 We also denote by c and δ the containment and protection parameters after a complete lock-down as it happened in France on March 17 (c =1 − (1 − δ)2).
The full description of the ODE model is given in the Appendix, a simplified version of which is presented in Figure 1.
Simplified compartmental model used. Corresponding notations are given in the Appendix.
Estimation of key parameters
115 When fitting available data, the key parameters driving the simulation results are: the fraction p of individuals with mild symptoms if infected, the reduction of contact rates during lock-down c, and the numbers of people exposed/asymptomatic/infected at the beginning of lock-down, on March 17. In order to reduce the parameter space, we have estimated the total number of 120 people exposed/asymptomatic/infected and inferred the number in each state by using the fractions of the mean time spent in each category, as given in Table 1. These parameters were estimated by comparing model predictions to the actual data of ICU occupation from March 17 to April 23 in France.
Simulation parameters used with relevant 95% con_dence intervals.
Uniform draws for the reduction of contact rates c during lock-down, between 60 and 80%,13 led to very peaked estimates at about c = 69% regardless of p and the total number of people exposed/asymptomatic/infected.
Keeping this value for c, we used the Approximate Bayesian Computation method (ABC)37 to derive estimates and confidence intervals for the other two parameters of interest. The ABC method was implemented with the mean standard error as a distance function,38 and an acceptance rate of 20%, which corresponds to an error of about 1 250 beds on average over the 38 data points. We assumed the prior distributions to be independent with the following choices: Beta distribution for p with parameters 2 390, 11 · 29, fitting the mean and 95% confidence interval observed,14 and uniform distribution for the total number of people exposed/asymptomatic/infected on March 17th with range between 0 and 2 million.
Using personalised Risk Predictions
A policy based on simulations with a risk-extended epidemic model requires identifying individuals at highest risk of severity (or other risks) and correspondingly advising them to remain in strict isolation, while relaxing isolation restrictions for individuals at lower risk. Such identification is done by risk- “scoring” models using common data science and machine learning techniques: logistic regression, random forest, and the likes. A standard metric to assess the discriminating power of such models is the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) curve.39
Combining the model’s ROC curve, with the more strictly isolated fraction, 1 − ρ, and the prevalence of severe symptoms in the population, p, results in the false positive and false negative error rates qFP, qFN. When ρ is small, the number of negative predictions is small (most people are in strict isolation), but the number of false negatives is also small, meaning that the vast majority of people in low isolation conditions will not experience severe symptoms.
A key question is to select a tolerable fraction of people (ρ) being submitted to lower isolation as a function of the performance of a risk prediction model so that to subject only few people to stricter isolation, yet not to violate the ICU bed capacity due to the errors made by the model in identifying such people. Technical details are provided in the Appendix.
Summary of Key Parameters and Data
The simulations were run with SEAIR parameters for the mean time spent in different phases of the disease, the basic reproduction number 
, and the transmission rate β, as listed in Table 1.
For the purpose of illustration, the class-conditional distributions w.r.t. to high/low risk were modelled with Beta distributions. Risk predictions, ROC curves and AUC were derived accordingly.
Parameters for the initial conditions S0, E0, A0, I0, U0 and R0 were taken depending on the investigated scenario: either one with “day 0” of March 17 – the first day of country-wide lock-down in France, or one with “day 0” of May 11 – the announced day for the beginning of the post lock-down exit.
3. Results
Figure 2 displays the number of individuals requiring an ICU bed w.r.t. time t. The March 17 scenario is in the left column, the May 11 scenario is in the right. Two risk models are considered: one with AUC of 95 · 99% (top row) and another, with AUC of 75 · 71% (bottom row), bracketing the performance of initial risk models developed for COVID-19.11,12
Number of individuals requiring an ICU bed w.r.t. time t (days). Left column starts on March 17 (the day of the initial lock-down in France), right column starts on May 11 (the day when lock-down ends). The dotted line on the left column shows the actual data for France from March 17 to April 23. Top row uses a risk prediction model with AUC 95 · 99%, bottom row uses a risk prediction model with AUC 75 · 71%.
In each plot, ρ represents the maximal percentage of the population that 175 is submitted to lighter restrictions (δr =0 · 1) in such a way that the 95% confidence interval of the number of individuals requiring an ICU bed when using the risk prediction model (green and orange curves) remains below the maximum number of ICU beds assumed (7 250). In these first simulations, the rest of the population is confined with stricter restrictions, δc = 0 · 9. Finally, the red curves show the number of individuals requiring an ICU bed w.r.t. time if the same ρ of population is in lower isolation, but selected at random without any risk prediction model.
Figure 2 shows that a high-AUC model (green curve) allows for having 61% in low isolation (δr =0 · 1, corresponding to a decrease of social interaction by 47%) from March 17 on, while a low-AUC model (orange curve) enables only 51%. In France, with a population of 67 million, these percentage differences correspond to 6 700 000 people. Plots for lock-down exit strategies (May 11) investigate the effect of the same risk prediction accuracies. All differences (61% vs 51% for March 17, 72% vs 60% for May 11) are statistically significant at the 5% level. Lastly, without a risk prediction model, the ICU beds demand greatly exceeds the current capacity at either ρ.
Figure 3 presents sensitivity analysis of the difference between the maximal percentage of people which may be in low isolation without exceeding ICU capacity for several risk-prediction models, relative to the same maximal percentage, but with no risk-prediction model. The results are shown for scenarios both for March 17 and for May 11. Sensitivity is tested with respect to the discrimination performance of the risk prediction models used for the risk-extended SEAIR simulations and the degree of isolation of the more strictly isolated population. We also alter the degree of isolation for the less strictly isolated population across different plots.
Maximal number of people in the low isolation group without exceeding the limit of 7 250 beds, with a margin of 2 000 beds as imposed by typical 95% confidence intervals, compared to the case of not using a risk prediction model. Plotted as a function of the AUC of a risk prediction model and the protection level δc for people recommended to be in isolation with stricter restrictions.
As expected, the higher the discrimination of the prediction model, the bigger the difference. The degree of isolation restrictions has different effects depending on who is considered: for the more strictly isolated population (the higher risk one) the stricter the isolation (parameter δc) the larger the impact of the risk prediction model. For the less strictly isolated population (the lower risk one − δr = 0 · 1 or 0 · 2 in Figure 3), the results are more intricate. It is also better to isolate more strictly, except when the risk prediction model is of very high quality and people of high risk are in stricter isolation. It is therefore important to both assume in models and encourage in practice (for example by focusing distribution of masks and other resources, strictly isolating nursing homes, etc.) realistic isolation practices for the high risk population.
Sensitivity analyses regarding the percentage p of the population with severe symptoms upon infection as well as the number of ICU beds available are included in the Appendix. As expected, the lower p the less the impact of a risk prediction model keeping AUC constant: given the limited ICU – and possibly other – resources, larger p allows for a smaller range of percentages of the population being under less strict isolation restrictions, making all differences between policies smaller in absolute terms. On the other hand, the more ICUs available the larger the impact of using a risk-prediction model, keeping everything else constant. More available resources allow for a larger range of percentage of people in less strict isolation making the differences between policies – risk based vs not – larger in absolute terms. Note that in all cases a risk prediction model approach allows for less people in strict isolation: this is consistent with value of information related arguments, as any test provides information which can be beneficial assuming everything else (including behavioural aspects) kept constant.
Finally, we explored examples of gradual shift of the population from stricter to less strict isolation restrictions. In the simulations, using the risk-enhanced SEAIR model and gradually releasing the population from the lowest to the highest predicted risk individuals allowed for a gradual increase of immunity in the population (results in Appendix) in a safer way than following the same policy without any risk prediction model.
Figure 4 shows example gradual policies, designed to ensure that the 95% confidence interval of the simulated infections does not exceed the ICU constraint. As in Figure 2, confidence intervals are computed by sampling according to the posterior distribution for p and the total number of people who had been infected on March 17. The insights complement those for single release policies. Simulations indicated that with risk-prediction models, a smaller percentage of the population may need to be subject to strict isolation policies. One could also reach the moment when isolation measures could be lifted sooner. For example, considering the May 11 exit using the high-AUC model and without exceeding the ICU capacity at any point, per Figure 4b, 72% of population could be in low isolation on ”day 0” (May 11), followed by another 25% on ”day 70” (July 20), and the remaining 3% on ”day 120” (September 8). Implementing the same exit schedule without a model would lead to ICU demand of over 25 000 beds. In contrast, a capacity-abiding exit strategy without a model (Figure 4f) would put only 51% of population in low isolation on ”day 0” (May 11), another 11% on ”day 80” (July 30), additional 18% on ”day 180” (November 7), and the last 20% only on ”day 270” (February 5, 2021), – or 5 months later than the similar risk-model-based strategy. A similar pattern is observed for planning the initial lock-down on March 17, and the insight qualitatively remains when a lower-quality prediction model is used. For both scenarios, simulation of the percentage of the population that becomes immune over time are shown in the Appendix. Because model-based policies release larger portions of the low-risk population and do so faster, they also achieve the levels of ”herd immunity” 
faster, allowing for the release of the high-risk population earlier as well. In other words, assuming everything else constant, the simulations indicated that isolation restriction may be relaxed faster or safer using risk prediction models, when such policies are considered.
Examples of gradual schedules of relaxing isolation restrictions with model-based risk predictions. Vectors t = [ti,t2] give the population release schedules ρ = [ρ1,ρ2] as follows: ρ1 · 100% of the population is released on day 0, then (ρ2 − ρ1) · 100% are released on day t1, etc. The left column starts with day 0 on March 17, right column starts with day 0 on May 11. The ”No model” lines in sub-figures (a) - (d) correspond to situations where the schedule is constructed based on the risk prediction model, but is implemented without the model, and in sub-figures (e) and (f) they correspond to situations where the schedule is constructed without a risk prediction model.
4. Discussion
Data-driven prediction models, which made large impacts in many areas the past decades, can enable, among others, personalisation of policies for managing epidemic outbreaks. We studied how prediction models for the severity of symptoms upon infection could be used in epidemic simulations to study the effect of non-pharmaceutical policies, particularly isolation restrictions, during an outbreak. We used COVID-19 data from France as of mid April 2020 as an example, and provided sensitivity analyses to understand how different parameters could impact epidemic simulations.
Simulations indicated that considering differential relaxation of isolation restrictions depending on predicted severity risk can decrease the percentage of the population in France under strict restrictions by 10% or more relative to not using such risk predictions, even when a moderate quality prediction model is available (e.g., AUC below 0 · 8). This result was robust to changes in risk prediction accuracy, percentage of severe-if-infected cases in the population, availability of resources (such as ICUs), and social distancing. Benefits increased when risk prediction accuracy increased, percentage of severe-if-infected cases in the population decreased, availability of resources (such as ICUs) increased, and social distancing increased. All results were developed using hypothetical risk prediction models for COVID-19, with discrimination ranges in line with early indications from initial models developed as of mid April 2020.11,12
The proposed approach can also be adopted for other epidemic models, and personalisation can further be explored using this approach for policies other than isolation restrictions. Moreover, predicted risk based isolation restrictions can be combined with other policies such as test-based ones, 5, 6 possibly also using other relevant prediction models, to limit the impact of outbreaks such as COVID-19.
Several caveats should be noted. First, epidemic models – and the conclusions they may support – rely on a number of parameters, for example virus incubation and recovery times and the basic reproduction number 
, while the effects of policies also depend on healthcare system factors such as the availability of relevant resources (e.g., trained personnel), or behavioural aspects of medical personnel and of the population. Second, the quality of a prediction model was assessed using AUC; other measures may be used for ranking based on risk.40 Third, making policy decisions using either standard models or risk-enhanced models studied in this work require careful context-specific robustness analysis. However, using risk prediction models can at worst make no significant difference while at best improve policies by a significant margin, with appropriate behavioural parameters. Finally, risk-predictions based policies using epidemic simulations should be developed taking into account behavioural aspects that may prove any model predictions wrong.
In conclusion, combining prediction models using data and data science and machine learning principles may improve outbreak management policies and should be considered when developing isolation policies. We studied how the benefits of these models depend on a number of outbreak, model, and behavioural parameters. As governments are considering relaxing isolation policies for COVID-19 in April 2020, models that use already known factors that affect the severity of symptoms of COVID-19 patients may prove useful.
Data Availability
No data. Code is available at https://reine.cmla.ens-cachan.fr/boulant/seair
Funding
No funding was used for this research.
Competing interests
No competing interests.
Data availability
Not Applicable.
Code availability
Code for this research is available at https://reine.cmla.ens-cachan.fr/boulant/seair
Correspondence
Correspondence should be addressed to TE (theodoros.evgeniou@insead.edu) and AO (anton.ovchinnikov@insead.edu) for general questions and to CP (camille.pouchol@parisdescartes.fr) for questions regarding code and methods.
Contributions
All authors contributed equally.
Acknowledgements
The authors are grateful to Ramsès Djidjou-Demasse for detailed exchanges about the model of his team,36 and to Amaury Lambert and Pierre-Yves Massè for the interesting discussions. We also thank Olivier Boulant for his help in making the code available on GitHub.
Appendix A. ODE model
Appendix A.1. Complete version
We first break down the 5 compartments S, E, A, I and R in 2 subcategories, depending on whether people are going to have mild (”m”) or severe (”s”) symptoms requiring ICU, upon infection. An added compartment for people in the ”s” category is that of people in ICU, denoted U.
These people are also labelled depending on whether they are considered low-risk and submitted to low isolation (”r”), or are considered high-risk and recommended to be in high isolation (”c”).
The notations are as follows: for a given category Q ∈ {S, E, A, I, U, R}, we use superscripts for risk prediction, and subscripts for actual status. In other words:
: low risk and submitted to low isolation restrictions, i.e. true negative.- : low risk and submitted to high isolation recommendations, i.e. false positive.
- : high risk and submitted to low isolation restrictions, i.e. false negative.
- : high risk and submitted to high isolation recommendations, i.e. true positive.
We also denote in category Q ∈ {S, E, A, I, U, R}
- : submitted to low isolation restrictions,
- : submitted to high isolation recommendations,
- : having mild symptoms if infected,
- : requiring ICU if infected,
- : total number of people in category Q.
We define the effective number of contagious people as
The equations read
The number of deaths is obtained by
Appendix A.2. Compact version
Although all compartments are needed, whether it is for the purpose of computations (see the definition of Ie) or for tracking the numbers in each category, the model may be written in a more compact form for convenience:
The number of deaths is obtained by
Computed parameters. The initial number of susceptibles S0 is computed as follows:
and the transmission rate β (computed in a situation without any policy aiming at reducing contacts and at the very beginning of the epidemic) is obtained through the following formula:
which follows from stability analysis, after neglecting the terms in 1 − p since p is very close to 1.36
Appendix A.3. Simulations
All simulations were run using Python. The number of time discretisation points per day was fixed at 500.
A given strategy of gradually relaxing restrictions is defined by N fractions of individuals put in the low-risk group
together with the times at which the policy changes
with T0 = 0 and T = tn a final horizon of interest.
The corresponding solution
was computed up until the final time T - further details below.
Each ρi uniquely determines how many people will be predicted to be in high or low risk, through
Assume that the solution at time Ti has been computed for a given i. Since the number of individuals in the low-risk group changes at time Ti, the ODE must be integrated for a corrected initial condition on the interval [Ti,Ti+1].
This initial condition was obtained from reallocating people depending on their new labelling. Denoting yold and ynew the value of y(Ti) before and after relabelling, the new value is given as a function of the previous one by
Appendix A.4. Further details on Figure 2 and 4
The number of samples for the prior distribution for p and the initial number of exposed/asymptomatic/infected at the beginning of lock-down E0 + A0 + I0 was set at n = 10 000, which leads to 2 000 posterior samples as the acceptance rate was at 20%.
In all figures showing the evolution of the number of people in ICU, the initial condition E0 + A0 +I0 and the proportion of people p not requiring ICU were sampled according to their posterior distribution.
Subsequent 95% confidence intervals were derived by removing the 2 · 5% and 2·5% upper and lower values for the computed number of ICU beds, respectively.
Appendix A.5. Synthetic risk prediction distributions
For all figures, the risk prediction was obtained from synthetic data. Distributions for people having critical symptoms requiring ICU and people having milder symptoms were assumed to follow Beta-distributions.
More precisely, denoting am, bm and as, bs the parameters for the respective distributions for people with mild symptoms and for people with severe symptoms, we fixed bs = am = 2 and made the other two parameters vary from as = bm = 2 (equivalent to not using a risk prediction model), with a maximum at as = bm = 6 · 5 used for Figure 3.
In Figure 2, no model refers to as =bm = as = bs = 1, low AUC model refers to bs = am = 2, as = bm = 3 and high AUC model refers to bs = am = 2, as = bm = 5.
Appendix A.6. Further details on Figure 3
Since Figure 3 is obtained by grid search along the variables δc and the risk prediction AUC, we did not sample according to the posterior distribution for each scenario, but rather computed mean values in order to ease the computational burden.
March 17: For March 17, simulations are obtained as follows:
the initial condition for ICU beds is known, and we used the estimate for the total number of infected people at that date. With our notations, the number E0 + A0 + I0 + R0 is about 2.6.106 people.14 We only needed an estimate for E0 + A0 + I0 and we did so by averaging over the posterior for this variable,
p was taken as the mean along posterior samples,
the ODE was then integrated on T = 200 days.
May 11: For May 11, simulations were obtained as follows:
sampling according to the posterior for p and E0 + A0 + I0, and integrating the ODE system from March 17 to May 11, we obtained a sample of initial conditions for May 11, of which we took the average,
the ODE was then integrated on T = 200 days.
Appendix B. Risk Prediction Models
The purpose of this section is to provide details about the practical use of ROC curves to calibrate the discrimination threshold in the personalised risk predictions. The key objective is to target the individuals at highest risk and apply differential policies to protect them from the risk of infection. Assuming that there is a constraint on the resources (for instance the number of ICU beds) and that even without constraints intensive care may not prevent death of all patients admitted (mortality rates for patients with mechanical ventilation may be higher than 50%), the questions underlying the implementation of large-scale risk predictions are the following: (i) how to rank the population with respect to their risk (e.g., of being eventually transferred to ICU), (ii) how to select the threshold value on this risk in order to further prevent individuals deemed at high risk being in contact with other individuals who may infect them, (iii) how sensitive will these estimates (ranking, threshold) be w.r.t. to sampling bias and to prediction accuracy. Here we only discuss the practical aspects related to (i) and (ii), and briefly comment on the sources of uncertainty (iii).
Percentage of people in the ’R’ compartment – immune – over time for the policy examples in Figure 4. Left is for policies starting on March 17th (Figures 4(a, c, e)); right is for policies starting on May 11th (Figures 4(b, d, f)). Note, ”herd immunity” is achieved for March 17th (May 11th) at 
, which happens at approximately day 170 (140) with the high-AUC model, day 300 (300) with low-AUC model, and not until day 400 (370) when no prediction model is used.
Building a personalised risk predictions. Personalised risk predictions are build upon past data which correlate individually a vector X of known factors with the outcome Y materialising the risk (e.g. the patient needing to be transferred to the intensive care unit, ICU) which is represented as a binary event (Y = +1 is a positive instance vs. Y = −1 is a negative instance). Such data are usually obtained after clinical studies. The fraction of high risk individuals in the population is denoted by p = P(Y = +1). Typical examples of methods to estimate personalised risk predictions are, for example, logistic regression in parametric statistics, or random forests, regularisation methods, or bipartite ranking algorithms in machine learning. In order to compare different estimation strategies, it is standard to assess performance using the ROC curve,39 defined as the parametric curve which maps, for a given risk prediction level s, each threshold value t to a point in the unit square with coordinates (P(s(X) > t | Y = −1), P(s(X) > t | Y = +1)) (plot of the true positive rate against the false positive rate).
Maximal number of people in the low isolation group the limit of 15 000 beds, with a margin of 2 000 beds as imposed by typical 95% confidence intervals, compared to the baseline case of not using a risk prediction model. Plotted as a function of the AUC of the risk prediction model used, and the protection level δc for people recommended to be in isolation. δr = 0 · 1 for all figures.
Discriminating high risk individuals. personalised risk predictions can be used to discriminate high versus low risk individuals based on the knowledge of individual values of relevant factors such as comorbidities. The decision is binary and it is taken after having calibrated the threshold t with respect to control parameters of a policy which are essentially driven by the constraints on the resources. Such constraints are for example: (a) the ICU capacity, which induces a constraint on the precision P(Y = 1 | s(X) > t) and such quantity is directly related to the true positive rate thanks to the Precision-Recall curve, or (b) the economic and psychological consequences of isolation measures on the population, which gives a constraint on P(s(X) > t).41
Sources of uncertainty. The previous estimates (risk predictions, decision threshold) are subject to uncertainties which can be due to sampling bias (e.g. clinical trial data used are not reflecting population data or electronic medical records failing to account for the part of the population who was never admitted to the hospital) or methodological bias (model misspecification, suboptimal machine learning/statistical method used). In order to provide statistical guarantees on the estimators obtained, it is necessary to compute confidence bands on the estimated ROC curve which will then lead to explicit confidence bands on the decision parameters of the risk prediction strategy. Typical approaches to derive confidence bands are to perform error propagation on distribution parameters (in a parametric framework), or to generate several ROC curves and Precision-Recall curves through resampling and provide some bootstrap estimate of the confidence band.42 Resampling strategies may include label flipping (prediction uncertainty), sample perturbation or shifting (sampling bias).
