An ensemble n-sub-epidemic modeling framework for short-term forecasting epidemic trajectories: Application to the COVID-19 pandemic in the USA

We analyze an ensemble of n-sub-epidemic modeling for forecasting the trajectory of epidemics and pandemics. These ensemble modeling approaches, and models that integrate sub-epidemics to capture complex temporal dynamics, have demonstrated powerful forecasting capability. This modeling framework can characterize complex epidemic patterns, including plateaus, epidemic resurgences, and epidemic waves characterized by multiple peaks of different sizes. We systematically assess their calibration and short-term forecasting performance in short-term forecasts for the COVID-19 pandemic in the USA from late April 2020 to late February 2022. We compare their performance with two commonly used statistical ARIMA models. The best fit sub-epidemic model and three ensemble models constructed using the top-ranking sub-epidemic models consistently outperformed the ARIMA models in terms of the weighted interval score (WIS) and the coverage of the 95% prediction interval across the 10-, 20-, and 30-day short-term forecasts. In the 30-day forecasts, the average WIS ranged from 377.6 to 421.3 for the sub-epidemic models, whereas it ranged from 439.29 to 767.05 for the ARIMA models. Across 98 short-term forecasts, the ensemble model incorporating the top four ranking sub-epidemic models (Ensemble(4)) outperformed the (log) ARIMA model 66.3% of the time, and the ARIMA model 69.4% of the time in 30-day ahead forecasts in terms of the WIS. Ensemble(4) consistently yielded the best performance in terms of the metrics that account for the uncertainty of the predictions. This framework could be readily applied to investigate the spread of epidemics and pandemics beyond COVID-19, as well as other dynamic growth processes found in nature and society that would benefit from short-term predictions.

individual top-ranking sub-epidemic models and the ARIMA models based on standard 1 0 1 performance metrics that account for the uncertainty of the predictions. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted June 21, 2022. ;https://doi.org/10.1101https://doi.org/10. /2022 We used daily COVID-19 deaths reported in the USA from the publicly available data tracking . Hence, WIS can be interpreted as a measure of The best fit sub-epidemic model and three ensemble models constructed using the top-ranking 3 5 8 sub-epidemic models (Ensemble(2), Ensemble(3), Ensemble (4)) yielded similar quality fits to 98 3 5 9 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted June 21, 2022. ;https://doi.org/10.1101https://doi.org/10. /2022 sub-epidemic models fit the data well, each of them results from the aggregation of two sub-3 8 0 epidemics characterized by different growth rates, scaling of growth, and outbreak sizes as  capture well the entire epidemic curve, including the latter plateau dynamics, by considering 3 8 6 models with two sub-epidemics. The best model fit (solid red line) and 95% prediction interval center panels, where the red and blue curves represent the two sub-epidemics and the grey curves 3 9 0 are the estimated epidemic trajectories. For each model fit, the residuals are also shown (right 3 9 1 panels). Black circles correspond to the data points.  panels) were derived for the top-ranking sub-epidemic model after fitting the sub-epidemic 3 9 5 modeling framework to the daily curve of COVID-19 deaths in the USA from 27-Feb-2020 to 3 9 6 20-April-2020 (see also Figure 2). Parameter estimates for both sub-epidemics are well 3 9 7 identified, as indicated by their relatively narrow bootstrap confidence intervals. Short-term forecasting performance 4 0 0 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The best fit sub-epidemic model and three ensemble models constructed using the top-ranking 4 0 2 sub-epidemic models (Ensemble(2), Ensemble(3), Ensemble(4)) consistently outperformed the 4 0 3 ARIMA models in terms of the weighted interval score (WIS) and the coverage of the 95% 4 0 4 prediction interval across the 10, 20 and 30 day short-term forecasts (Table 2). For instance, for 4 0 5 30-day forecasts, the average WIS ranged from 377.6 to 421.3 for the sub-epidemic models,  Figure 6). Similarly, the coverage of the 95% PI ranged from 82.2% to 88.2% for the sub-4 1 0 epidemic models, whereas it ranged from 58% to 60.3% for the ARIMA models in 30-day 4 1 1 forecasts. In terms of the coverage of the 95% PI, the Ensemble(4) outperformed the (log)  described in the text.  . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Ensemble(4) model outperformed the (log) ARIMA model. For example, the Ensemble(4)   In terms of the metrics based on point estimate information, the ARIMA models showed lower Ensemble(4) achieved the best forecasting performance in 30-day forecasts (Table 2). Overall,  Representative 30-day forecasts of the top-ranking sub-epidemic models to the daily curve of . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted June 21, 2022. ; https://doi.org/10. 1101/2022 period. For instance, the top-ranked sub-epidemic model predicts a decline in the mortality  the start time of the forecast. Circles correspond to the data points. These four top-ranking 4 6 0 models support forecasts with diverging trajectories even though they yield similar fits to the 4 6 1 calibration period. For instance, the 1 st ranked sub-epidemic model predicts a decline in the In sensitivity analyses, defining ensemble weights as proportional to the relative likelihood did 4 7 9 not achieve better performance relative to the ensemble models generated using weights 4 8 0 proportional to the reciprocal of the AIC c . Moreover, the rank of the ensemble models was not affected by the type of weights (Table 3).  . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted June 21, 2022.  were based on the relative likelihood) and the ARIMA models across 98 sequential weekly T h  e  c  h  a  l  l  e  n  g  e  s  o  f  m  o  d  e  l  i  n  g  a  n  d  5  9  1   f  o  r  e  c  a  s  t  i  n  g  t  h  e  s  p  r  e  a  d  o  f  C  O  V  I  D  -1  9 . I  D  :  3  2  6  1  6  5  7  4  ;  P  u  b  M  e  d  C  e  n  t  r  a  l  5  9  3   P  M  C  I  D  :  P  M  C  P  M  C  7  3  8  2  2  1  3  .  5  9  4   2 .  . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted June 21, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted June 21, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted June 21, 2022. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted June 21, 2022. ; https://doi.org/10.1101/2022.06.19.22276608 doi: medRxiv preprint 3 4 5 6 . R  g  u  i  b  i  M  A  ,  M  o  u  s  s  a  N  ,  M  a  d  a  n  i  A  ,  A  a  r  o  u  d  A  ,  Z  i  n  e  -D  i  n  e  K  .  F  o  r  e  c  a  s  t  i  n  g  C  o  v  i  d  -1  9  7  6  2   T  r  a  n  s  m  i  s  s  i  o  n  w  i  t  h  A  R  I  M  A  a  n  d  L  S  T  M  T  e  c  h  n  i  q  u  e  s  i  n  M  o  r  o  c  c  o  .  S  N  C  o  m  p  u  t  S  c  i  .  2  0  2  2  ;  3  (  2  )  :  1  3  3  -.  7  6  3   E  p  u  b  2  0  2  2  /  0  1  /  1  4  .  d  o  i  :  1  0  .  1  0  0  7  /  s  4  2  9  7  9  -0  2  2  -0  1  0  1  9  -x  .  P  u  b  M  e  d  P  M  I  D  :  3  5  0  4  3  0  9  6  .  7  6  4   5  7  .  K  a  n  d  u  l  a  S  ,  S  h  a  m  a  n  J  .  N  e  a  r  -t  e  r  m  f  o  r  e  c  a  s  t  s  o  f  i  n  f  l  u  e  n  z  a  -l  i  k  e  i  l  l  n  e  s  s  :  A  n  e  v  a  l  u  a  t  i  o  n  o  f  7  6