Abstract
Background In early May 2020, following social distancing measures due to COVID-19, governments consider relaxing lock-down. We combined individual clinical risk predictions with epidemic modelling to examine simulations of risk-based differential isolation and exit policies.
Methods We extended a standard susceptible-exposed-infected-removed (SEIR) model to account for personalised predictions of severity, defined by the risk of an individual needing intensive care if infected, and simulated differential isolation policies using COVID-19 data and estimates in France as of early May 2020. We also performed sensitivity analyses. The framework may be used with other epidemic models, with other risk predictions, and for other epidemic outbreaks.
Findings Simulations indicated that, assuming everything else the same, an exit policy considering clinical risk predictions starting on May 11, as planned by the French government, could enable to immediately relax restrictions for an extra 10% (6 700 000 people) or more of the lowest-risk population, and consequently relax the restrictions on the remaining population significantly faster - while abiding to the current ICU capacity. Similar exit policies without risk predictions would exceed the ICU capacity by a multiple. Sensitivity analyses showed that when the assumed percentage of severe patients among the population decreased, or the prediction model discrimination improved, or the ICU capacity increased, policies based on risk models had a greater impact on the results of epidemic simulations. At the same time, sensitivity analyses also showed that differential isolation policies require the higher risk individuals to comply with recommended restrictions. In general, our simulations demonstrated that risk prediction models could improve policy effectiveness, keeping everything else constant.
Interpretation Clinical risk prediction models can inform new personalised isolation and exit policies, which may lead to both safer and faster outcomes than what can be achieved without such prediction models.
Funding No funding was used for this research.
Evidence before this study Several countries have implemented non-pharmaceutical interventions based on social distancing and isolation measures in order to limit the spread of COVID-19. There has been limited differentiation in the degree of isolation measures, except for those critical for the functioning of the healthcare system and other key services. There is limited evidence about the impact of relaxing these measures as this has happened only recently and in only a few countries. Investigating the potential impact of differential restrictions depending on medical factors, such as the risk of severe symptoms if infected by Sars-Cov-2, may inform policies for imposing or relaxing isolation policies when these are considered.
Added value of this study This study investigates incorporating clinical risk predictions in epidemic models, allowing to explore isolation policies that consider individual clinical risks using simulations.
Implications of all the available evidence Epidemic simulations of isolation policies that consider predicted clinical risks in order to differentiate restrictions indicate the feasibility of policies that may be otherwise impossible to implement without undertaking this type of risk-based approach.
1. Introduction
As of May 2020, many countries have adopted non-pharmaceutical interventions, such as isolation restrictions,1 to control the spread of COVID-19. Epidemic models have been used to inform such policies.2,3 Governments now consider relaxing these restrictions. Immunity tests4 and technology5 may support such policy decisions. We considered an application of predictive technologies, such as machine learning, that can be used to better understand outbreaks using epidemic simulations: using personalised predictions of severity risk, defined as requiring ICU if infected, in epidemic simulations to examine differential risk based isolation policies. Other clinical risks can be considered similarly.
We extended standard epidemic models, namely a version of SEIR,6 to incorporate personalised risk predictions. Using simulations, we investigated how prediction models for patient severity may inform policy in two scenarios. First, when there is an ongoing outbreak as it was the case in France on the 17th of March, 2020, when lock-down started. Second, when the outbreak has been curbed by lock-down and progressive loosening of isolation policies (”exit”) may take place, as planned in France starting from the 11th of May, 2020.
There has already been research indicating differential impact of COVID-19 across patients, for example depending on hypertension, diabetes or other factors,7 but there is currently no standard risk prediction model considering all factors, although some early versions are available.8,9 Therefore we assumed hypothetical risk prediction models and studied the sensitivity of the simulation results with respect to model discrimination.
To populate the models, we used available COVID-19 estimates and data from France as of early May 2020.10 At the time, there were about 4 000 beds occupied by people with COVID-19, with a peak at 7 148 people in intensive care, compared to the total roughly 10 000 capacity recently reached by the French health system. We used current estimates with a reproduction number value of 
prior to lock-down, and an estimate of 2·5 million people who had been immune or infected when it started on March 17.11
Numerous containment/lock-down strategies have been proposed to flatten the predicted curve of the number of severe cases by means of non-pharmaceutical interventions such as strict quarantine,12 social distancing, strict hygiene, population screening, etc.3, 13, 14, 15 Although social distancing might be applied to each individual in a similar way, a different policy consists of adjusting it to the seriousness of the symptoms one has if infected.16,17,18 However, severity of symptoms upon infection is not known a priori and, at best, can only be predicted. Such predictions may be possible using predictive models, for example using rules, data science, and machine learning methods and principles, based on data related to infection symptoms.8,9
2. Methods
2.1. Risk-Extended SEAIR Model
The model extends a standard compartmentalised SEAIR model with an added compartment for people in ICU by introducing four subcategories for each compartment. The risk model classifies each individual into the “high” versus “low” risk group, where risk refers to an individual experiencing severe symptoms requiring ICU.
Any prediction model makes mistakes. As a result, each individual would fall into exactly one of four groups which form a so-called “confusion matrix”: (a) he/she would actually need ICU bed upon infection and was classified as high-risk (true positive), (b) would need ICU bed upon infection but classified low-risk (false negative), (c) would not need ICU bed upon infection but classified high-risk (false positive), (d) would not need ICU bed upon infection and classified low-risk (true negative). As a whole, this led to 22 possible compartments in the extended SEAIR model.
For the control parameter of a policy applying differentiated isolation restriction w.r.t. to the predicted risk of developing severe symptoms we considered the proportion ρ of individuals subject to low isolation restrictions, i.e. those classified by the risk-model as the low-risk group. We denote with p the fraction of individuals who are actually in the low-risk group (that is, those who would actually have mild symptoms and not require ICU if infected.) Depending on the characteristics of the risk prediction model and on p and ρ, the false positive and false negative rates, denoted by qFP and qFN, vary per the following implicit function equation:
We assumed that the differentiated isolation restrictions change people’s behaviour per two behavioural parameters δr (for the group with low isolation restrictions, i.e. released) and δc (for the group with high isolation restrictions, i.e. confined); 0 < δr ≤ δc ≤ 1. Variables δr and δc capture a level of ‘‘protection”, which may aggregate several factors such as respiratory and hand hygiene, how much a person has lowered the number of exits from home and social interactions, etc. How individuals of a given category reduce their contacts not only depends on δc and δr but also on the proportion ρ of people with less strict restrictions. The decrease in contact rates, cr and cc for the two groups, may be obtained with the following formula:
These parameters have been used in the literature modelling the current lock-down.19,10 We also denote by c and S the containment and protection parameters after a complete lock-down as it happened in France on March 17 (c = 1-(1-δ)2).
The full description of the ODE model is given in the Appendix, a simplified version of which is presented in Figure 1.
Simplified compartmental model used. Corresponding notations are given in the Appendix.
2.2. Estimation of key parameters
When fitting available data, the key parameters driving the simulation results are: the fraction p of individuals with mild symptoms if infected, the 85 reduction of contact rates during lock-down c, and the numbers of people exposed/asymptomatic/infected at the beginning of lock-down, on March 17. These parameters were estimated by comparing model predictions to the actual data of ICU occupation from March 17 to May 2 in France.
We used the Approximate Bayesian Computation method (ABC)20 to derive estimates and confidence intervals for the three parameters of interest. The ABC method was implemented with the root mean standard error as a distance function,21 with a maximum error set at 1 000 beds on average over the 47 data points, which corresponds to an acceptance rate of about 13%. We assumed the prior distributions to be independent with the following choices: Beta distribution for p with parameters 2 390, 11 · 29, fitting the mean and 95% confidence interval observed,11 uniform distribution for the total number of people exposed/asymptomatic/infected on March 17 with range between 1 and 2 million, and uniform distribution for c between 65% and 75%.
2.3. Using personalised Risk Predictions
A policy based on simulations with a risk-extended epidemic model requires identifying individuals at highest risk of severity and correspondingly advising them to remain in strict isolation, while relaxing isolation restrictions for individuals at lower risk. Such identification is done by risk-“scoring” models using common data science and machine learning techniques: logistic regression, random forest, and the likes. A standard metric to assess the discriminating power of such models is the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) curve.22
Combining the model’s ROC curve, with the more strictly isolated fraction, 1 –ρ, and the prevalence of severe symptoms in the population, p, results in the false positive and false negative error rates qFP, qFN. When ρ is small, the number of negative predictions is small (most people are in strict isolation), but the fraction of false negatives is also small, meaning that the vast majority of people in low isolation conditions will not experience severe symptoms. Increasing ρ not only releases more people from isolation, but also increases the false negative rate. Technical details are provided in the Appendix.
A key question is to select a tolerable fraction of people (ρ) being submitted to lower isolation as a function of the performance of a risk prediction model so that to subject only few people to stricter isolation, yet not to violate the ICU bed capacity due to the errors made by the model in identifying such people.
2.4. Summary of Key Parameters and Data
All parameters used in simulations are listed in Table 1.
Simulation parameters used with relevant 95% confidence intervals.
For the purpose of illustration, the class-conditional distributions w.r.t. to high/low risk were modelled with Beta distributions. Risk predictions, ROC curves and AUC were derived accordingly.
Parameters for the initial conditions S0, E0, A0, I0, U0 and R0 were taken depending on the investigated scenario: in the first one the “day 0” is set on March 17 2020 - the first day of country-wide lock-down in France, while in the second one “day 0” is set on May 11 2020 - the announced day for the beginning of the post lock-down exit.
3. Results
Figure 2 displays the number of individuals requiring an ICU bed w.r.t. time t. The March 17 scenario is in the left column, the May 11 scenario is in the right. Two risk models are considered: one with AUC of 95 · 99% (top row) and another, with AUC of 75 · 71% (bottom row), bracketing the performance of initial risk models developed for COVID-19.8,9
Number of individuals requiring an ICU bed w.r.t. time t (days). Left column starts on March 17 (the day of the initial lock-down in France), right column starts on May 11 (the day when lock-down ends). The dotted line on the left column shows the actual data for France from March 17 to May 2. Top row uses a risk prediction model with AUC 95 · 99%, bottom row uses a risk prediction model with AUC 75 · 71%.
In each plot, ρ represents the maximal percentage of the population that is submitted to lighter restrictions (δr = 0-1) in such a way that the 95% confidence interval of the number of individuals requiring an ICU bed when using the risk prediction model (green and orange curves) remains below the maximum number of ICU beds assumed (7 250). In these first simulations, the rest of the population is confined with stricter restrictions, δc = 0 · 9. Finally, the red curves show the number of individuals requiring an ICU bed w.r.t. time if the same ρ of population is in lower isolation, but selected at random without any risk prediction model.
Figure 2 shows that a high-AUC model (green curve) allows for having 60% in low isolation (δr =0 · 1, corresponding to a decrease of social interaction by 47%) from March 17 on, while a low-AUC model (orange curve) enables only 51%. In France, with a population of 67 million, these percentage differences correspond to 6 700 000 people. Plots for lock-down exit strategies (May 11) investigate the effect of the same risk prediction accuracies. All differences (60% vs 51% for March 17, 70% vs 59% for May 11) are statistically significant at the 5% level. Lastly, without a risk prediction model, the ICU beds demand greatly exceeds the current capacity at either ρ.
Figure 3 presents sensitivity analyses of the difference between the maximal percentage of people which may be in low isolation without exceeding ICU capacity for several risk-prediction models, relative to the same maximal percentage, but with no risk-prediction model. The results are shown for both the March 17 and the May 11 scenarios. Sensitivity is tested with respect to the discrimination performance of the risk prediction models used for the risk-extended SEAIR simulations and the degree of isolation of the more strictly isolated population (δc). We also alter the degree of isolation for the less strictly isolated population (δr) across different plots.
Difference in maximum possible percentage of people in low isolation without hospital saturation. Maximal number of people in the low isolation group without exceeding the limit of 7 250 beds, with a margin of 2 000 beds as imposed by typical 95% confidence intervals, compared to the case of not using a risk prediction model. Plotted as a function of the AUC of a risk prediction model and the protection level δc for people recommended to be in isolation with stricter restrictions. p ~ 0 · 995 for all figures.
As expected, the higher the discrimination of the prediction model, the bigger the difference. The degree of isolation restrictions has different effects depending on who is considered: for the more strictly isolated population (the higher risk one) the stricter the isolation (parameter δc) the larger the impact of the risk prediction model. For the less strictly isolated population (the lower risk one - δr =0 · 1 or 0 · 2 in Figure 3), the results are more intricate. It is also better to isolate more strictly, except when the risk prediction model is of very high quality and people of high risk are in stricter isolation. It is therefore important to both assume in models and encourage in practice (for example by focusing distribution of masks and other resources, strictly isolating nursing homes, etc.) realistic isolation practices for the high risk population.
Figure 4 shows sensitivity analyses regarding the percentage p of the population 175 with severe symptoms upon infection, if we were to only decrease (from 0 · 995 to 0 · 98) the value for p while following the same analysis as before. The results are shown with a different ICU constraint (15 000 instead of 7 500) than in Figure 3 to also study how the ICU constraint affects the conclusions.
Difference in maximum possible percentage of people in low isolation without hospital saturation. Maximal number of people in the low isolation group without exceeding the limit of 15 000 beds, with a margin of 2 000 beds as imposed by typical 95% confidence intervals, compared to the case of not using a risk prediction model. Plotted as a function of the AUC of a risk prediction model and the protection level δc for people recommended to be in isolation with stricter restrictions. δr = 0 · 1 for all figures.
First, as expected, the lower p the less the impact of a risk prediction model 180 keeping AUC constant: given the limited ICU - and possibly other - resources, larger p allows for a smaller range of percentages of the population being under less strict isolation restrictions, making all differences between policies smaller in absolute terms. Second, when we compare Figures 3 (a) and (c) with Figures 4 (a) and (c), where the only difference is in the total ICU assumed (7 500 vs 15 000), we see that the more ICUs available the larger the impact of using a risk-prediction model, keeping everything else constant. More available resources allow for a larger range of percentage of people in less strict isolation making the differences between policies - risk based vs not - larger in absolute terms. Note that in all cases a risk prediction model approach allows for fewer people in strict isolation: this is consistent with value of information related arguments, as any test provides information which can be beneficial assuming everything else (including behavioural aspects) kept constant.
Finally, we explored the implications of our simulation for gradual exit strategies. To consider practical and realistic scenarios we solved the resultant optimisation (dynamic program) allowing releases every 30 days at multiples of 5% of the population, while ensuring that the maximum number of ICU beds needed was at most 5 250, being at least 2 000 beds (the typical maximum confidence range in Figure 2) below the maximum ICU capacity assumed.
Table 2 shows the total number of months to release the entire population for different scenarios, keeping all other parameters constant: no model, low-AUC model, high-AUC model, and for 3 different values of δc. We considered only gradual releases in 3 or 4 waves - the ICU system was overwhelmed when using only 2 waves for most simulations. The main insight is that using no risk model would require more than a year in all scenarios, while exit based on risk-based models would lead to relaxing restrictions for the entire population as fast as over 5 (March 17) or 4 (May 11) months.
Minimal time (in months) required for all people to exit high isolation restrictions, starting from March 17 (upper tables) or May 11 (lower tables), depending on δc, model quality and the number of waves of gradual relaxations. δr = 0 · 1
Figure 5 shows example gradual policies for May 11 corresponding to the four waves part of Table 2, and assuming δc = 0 · 9 as in Figure 2. The insights complement those for single release policies: with risk-prediction models, a smaller percentage of the population may need to be subject to stricter isolation policies. Consequently, one could also reach the moment when isolation measures could be lifted sooner.
Examples of gradual schedules of relaxing isolation restrictions with and witout model-based risk predictions. High AUC model (green) and no model (red) ρ = [0 · 7, 0 · 75, 0 · 9, 1] and t = [60, 90, 120, 600], no model (blue): ρ = [0.5, 0·6, 0·7,1] and t = [90, 180, 330, 600]. Vectors t = [t1,t2] give the population release schedules ρ = [ρ1, ρ2] as follows: ρ1 · 100% of the population is released on day 0, then (ρ2 – ρ1) · 100% are released on day t1, etc.
For example, using the high-AUC model and without exceeding the ICU capacity at any point, 70% of the lowest-risk population could be released on May 11 (“day 0”), followed by another 5% on July 10 (“day 60”), and yet another 15% on August 9 (“day 90”), and finally releasing the remaining 10% on September 8 (“day 120”). The resultant ICU demand is shown as a green line on Figure 5a. Implementing the same exit schedule without a model would lead to ICU demand of over 25 000 beds (red line). In contrast, a capacity-abiding exit strategy without a model (blue line) would put only 50% of population in low isolation on “day 0” (May 11), another 10% on “day 90” (August 9), additional 10% on “day 180” (November 7), and the last 30% only on “day 330” (April 6, 2021), - 7 months later than the similar risk-model-based strategy. Such an extended isolation would also apply to many more people: 10% with the model versus 30% without; for France this means the additional ~13.5 million people in isolation for the additional 7 months.
For both scenarios, Figure 5b shows the percentage of the population that becomes immune over time. Because model-based policies release larger portions of the low-risk population and do so faster, they also achieve the levels of “herd immunity” 
, allowing for the ultimate protection against the disease. In contrast, herd immunity is not achieved by a policy without the risk-model: the disease is suppressed, but could explode again. In other words, assuming everything else constant, the simulations indicated that, using risk prediction models, isolation restriction may be relaxed faster and safer.
4. Discussion
Simulations indicated that considering differential relaxation of isolation restrictions depending on predicted severity risk can decrease the immediate percentage of the population in France under strict restrictions by 10% or more relative to not using such risk predictions, and fasten the complete exit by several months. This result was robust to changes in risk prediction accuracy, percentage of severe-if-infected cases in the population, availability of resources (such as ICUs), and social distancing. Benefits increased when risk prediction accuracy increased, percentage of severe-if-infected cases in the population decreased, availability of resources (such as ICUs) increased, and social distancing increased. All results were developed using hypothetical risk prediction models for COVID-19, with discrimination ranges in line with early indications from initial models developed as of early May 2020.8,9
The proposed approach can also be adopted for other epidemic models, and personalisation can further be explored using this approach for policies other than isolation restrictions. Moreover, predicted risk based isolation restrictions can be combined with other policies such as test-based ones,4,5 possibly also using other relevant prediction models, to limit the impact of outbreaks such as COVID-19. Finally, the same analysis can be done focusing on mortality or other outcomes instead of ICU demand.
Several caveats should be noted. First, epidemic models - and the conclusions they may support - rely on a number of parameters, for example virus incubation and recovery times and the basic reproduction number 
, while the effects of policies also depend on healthcare system factors such as the availability of relevant resources (e.g., trained personnel). Second, these parameters are uncertain and evolve dynamically;23 the resultant policies are therefore contingent. Observing an ICU demand that is closer to an upper boundary of the confidence interval may require the next wave to be delayed or involve a smaller release percentage than out current simulations built from day zero suggest. Third, policy decisions require careful context-specific robustness analysis; however, using risk prediction models can at worst make no significant difference while at best improve policies by a significant margin, fixing all other conditions. Fourth, risk prediction models cannot be used when, or for people, for whom the necessary data is unavailable. In this case, simple models (e.g., based only on age and some reliable chronic disease data) may need to be used, which may limit the benefits of the approach. Finally, risk-predictions based policies using epidemic simulations should be developed taking into account behavioural aspects that may prove any model predictions and policy actions wrong; ethical issues, fear, widespread non-compliance to isolation measures, and the likes.
In conclusion, combining prediction models using data science and machine learning principles may improve outbreak management policies and should be considered when developing isolation and exit policies.
Data Availability
No data. Code is available at https://reine.cmla.ens-cachan.fr/boulant/seair
Funding
No funding was used for this research.
Competing interests
No competing interests.
Data availability
Not Applicable.
Code availability
Code for this research is available at https://reine.cmla.ens-cachan.fr/boulant/seair
Correspondence
Correspondence should be addressed to TE (theodoros.evgeniou{at}insead.edu) and AO (anton.ovchinnikov{at}insead.edu) for general questions and to CP (camille.pouchol{at}parisdescartes.fr) for questions regarding code and methods.
Contributions
All authors contributed equally for the research and analysis. Camille Pouchol and Mathilde Fekom developed the code.
Acknowledgements
The authors are grateful to Ramsès Djidjou-Demasse for detailed exchanges about the model of his team,19 and to Amaury Lambert and Pierre-Yves Massé for the interesting discussions. We also thank Olivier Boulant for his help in making the code available on GitHub.
Appendix A. Extended SEAIR model
Appendix A.1. Complete version
We first break down the 5 compartments S, E, A, I and R in 2 subcategories, depending on whether people are going to have mild (”m”) or severe (”s”) symptoms requiring ICU, upon infection. An added compartment for people in the “s” category is that of people in ICU, denoted U.
These people are also labelled depending on whether they are considered low-risk and submitted to low isolation (”r”), or are considered high-risk and recommended to be in high isolation (”c”).
The notations are as follows: for a given category Q ϵ {S, E, A, I, U, R}, we use superscripts for risk prediction, and subscripts for actual status. In other words:
: would not require ICU if infected, submitted to low isolation restrictions, i.e. true negative,- : would not require ICU if infected, submitted to high isolation recommendations, i.e. false positive,
- : would require ICU if infected, submitted to low isolation restrictions, i.e. false negative,
- : would require ICU if infected, submitted to high isolation recommendations, i.e. true positive.
We also denote in category Q ϵ {S, E, A, I, U, R}
- : submitted to low isolation restrictions,
- : submitted to high isolation recommendations,
- : having mild symptoms if infected,
- : requiring ICU if infected,
Q = Qm + Qs = Q(r) + Q(c): total number of people in category Q.
We define the effective number of contagious people as
The equations read
The number of deaths is obtained by
Appendix A.2. Compact version
Although all compartments are needed, whether it is for the purpose of computations (see the definition of Ie) or for tracking the numbers in each category, the model may be written in a more compact form for convenience:
The number of deaths is obtained by
Computed parameters
The initial number of susceptibles S0 is computed as follows:
and the transmission rate β is given by the following formula:
which can be obtained by stability analysis.19 Here Sinit stands for the initial 305 number of susceptible people for the period during which 
has been estimated. As all estimates for 
have been obtained for the period prior to lock-down, we chose Sinit = N0 where N0 is the total number of people in the population of interest.
Appendix A.3. Simulations
All simulations were run using Python. The number of time discretisation points per day was fixed at 500.
A given strategy of gradually relaxing restrictions is defined by N fractions of individuals put in the low-risk group
together with the times at which the policy changes
with T0 = 0 and T = TN a final horizon of interest.
The corresponding solution
was computed up until the final time T - further details below.
Each ρi uniquely determines how many people will be considered to be high or low risk, through
Assume that the solution at time Ti has been computed for a given i. Since the number of individuals in the low-risk group changes at time Ti, the ODE must be integrated for a corrected initial condition on the interval [Ti,Ti+1].
This initial condition was obtained from reallocating people depending on their new labelling. Denoting yold and ynew the value of y(Ti) before and after relabelling, the new value is given as a function of the previous one by
Appendix A.4. Further details for Figure 2
The number of samples for the prior distribution for p and the initial number of exposed/asymptomatic/infected at the beginning of lock-down E0 + A0 + I0 was set at n = 10 000 (resp. 100 000 for robustness when computing means). This led to around 1 300 (resp. 13 000) posterior samples as the acceptance rate was at 13%. In order to reduce the parameter space, we have estimated the total number of people exposed/asymptomatic/infected and inferred the number in each state by using the fractions of the mean time spent in each category, as given in Table 1.
In all figures showing the evolution of the number of people in ICU, the initial condition E0 + A0 +I0 and the proportion of people p not requiring ICU were sampled according to their posterior distribution.
Subsequent 95% confidence intervals were derived by removing the 2 · 5% and 2·5% upper and lower values for the computed number of ICU beds, respectively.
Appendix A.5. Synthetic risk prediction distributions
For all figures, the risk prediction was obtained from synthetic data. Distributions for people having critical symptoms requiring ICU and people having milder symptoms were assumed to follow Beta-distributions.
More precisely, denoting am,bm and as, bs the parameters for the respective distributions for people with mild symptoms and for people with severe symptoms, we fixed bs = am = 2 and made the other two parameters vary from as = bm = 2 · 5, with a maximum at as = bm = 6 · 5 used for Figures 3 and 4. In Figures 2, 5 and Table 3, no model refers to am = bm = as = bs = 1, low 340 AUC model refers to bs = am = 2, as = bm = 3 and high AUC model refers to bs = am = 2, as = bm = 5.
Appendix A.6. Further details on Figures 3, 4, 5 and Table 3
Since Figures 3, 4, T and Table 3 require grid searches, we did not sample according to the posterior distribution for each scenario, but rather computed mean values in order to ease the computational burden.
March 17: For March 17, simulations are obtained as follows:
the initial condition for ICU beds is known, and we used the estimate for the total number of infected people at that date. With our notations, the number E0 + A0 + I0 + R0 is about 2.6.106 people.11 We only needed an estimate for E0 + A0 + I0 and we did so by averaging over the posterior for this variable,
p was taken as the mean along posterior samples,
the ODE was then integrated on T = 200 days.
May 11: For May 11, simulations were obtained as follows:
sampling according to the posterior for p and E0 + A0 +I0, and integrating the ODE system from March 17 to May 11, we obtained a sample of initial conditions for May 11, of which we took the average,
the ODE was then integrated on T = 200 days.
Figure 5 and Table 3: Table 3 computes the minimal possible time the whole population may be in low isolation, without breaking the ICU constraint of 7 250 beds with a 2 000 bed margin taking into account typical sizes of 95% confidence intervals. The minimal time was computed by searching for strategies where times of (gradual) relaxations are chosen every month, and percentages of people in low isolation are multiples of 5%. The initial percentage of people in low isolation is always chosen to be the highest possible.
Appendix B. Risk Prediction Models
The purpose of this section is to provide details about the practical use of ROC curves to calibrate the discrimination threshold in the personalised risk predictions. The key objective is to target the individuals at highest risk and apply differential policies to protect them from the risk of infection. Assuming that there is a constraint on the resources (for instance the number of ICU beds) and that even without constraints intensive care may not prevent death of all patients admitted (mortality rates for patients with mechanical ventilation may be higher than 50%), the questions underlying the implementation of large-scale risk predictions are the following: (i) how to rank the population with respect to their risk (e.g., of being eventually transferred to ICU), (ii) how to select the threshold value on this risk in order to further prevent individuals deemed at high risk being in contact with other individuals who may infect them, (iii) how sensitive will these estimates (ranking, threshold) be w.r.t. to sampling bias and to prediction accuracy. Here we only discuss the practical aspects related to (i) and (ii), and briefly comment on the sources of uncertainty (iii).
Building a personalised risk predictions
Personalised risk predictions are build upon past data which correlate individually a vector X of known factors with the outcome Y materialising the risk (e.g. the patient needing to be transferred to the intensive care unit, ICU) which is represented as a binary event (Y = +1 is a positive instance vs. Y = −1 is a negative instance). Such data are usually obtained after clinical studies. The fraction of high risk individuals in the population is denoted by p = P(Y = −1). Typical examples of methods to estimate personalised risk predictions are, for example, logistic regression in parametric statistics, or random forests, regularisation methods, or bipartite ranking algorithms in machine learning. In order to compare different estimation strategies, it is standard to assess performance using the ROC curve,22 defined as the parametric curve which maps, for a given risk prediction level s, each threshold value t to a point in the unit square with coordinates (P(s(X) > t | 395 Y = −1), P(s(X) > t | Y = +1)) (plot of the true positive rate against the false positive rate).
Discriminating high risk individuals
Personalised risk predictions can be used to discriminate high versus low risk individuals based on the knowledge of individual values of relevant factors such as comorbidities. The decision is binary and it is taken after having calibrated the threshold t with respect to control parameters of a policy which are essentially driven by the constraints on the resources. Such constraints are for example: (a) the ICU capacity, which induces a constraint on the precision P(Y = 1 | s(X) > t) and such quantity is directly related to the true positive rate thanks to the Precision-Recall curve, or (b) the economic and psychological consequences of isolation measures on the population, which gives a constraint on P(s(X) > t).24
Sources of uncertainty
The previous estimates (risk predictions, decision threshold) are subject to uncertainties which can be due to sampling bias (e.g. clinical trial data used are not reflecting population data or electronic medical records failing to account for the part of the population who was never admitted to the hospital) or methodological bias (model misspecification, suboptimal machine learning/statistical method used). In order to provide statistical guarantees on the estimators obtained, it is necessary to compute confidence bands on the estimated ROC curve which will then lead to explicit confidence bands on the decision parameters of the risk prediction strategy. Typical approaches to derive confidence bands are to perform error propagation on distribution parameters (in a parametric framework), or to generate several ROC curves and Precision-Recall curves through resampling and provide some bootstrap estimate of the confidence band.25 Resampling strategies may include label flipping (prediction uncertainty), sample perturbation or shifting (sampling bias). Lastly, the quality of a prediction model was assessed using AUC; other measures may be used for ranking based on risk.26
Footnotes
† Email addresses: mathilde.fekom{at}ens-paris-saclay.fr (Mathilde Fekom), anton.ovchinnikov{at}queensu.ca, (Anton Ovchinnikov), raphael.porcher{at}aphp.fr (Raphael Porcher), camille.pouchol{at}parisdescartes.fr (Camille Pouchol), nicolas.vayatis{at}ens-paris-saclay.fr (Nicolas Vayatis)
