Informing pandemic response in the face of uncertainty. An evaluation of the U.S. COVID-19 Scenario Modeling Hub

Our ability to forecast epidemics more than a few weeks into the future is constrained by the complexity of disease systems, our limited ability to measure the current state of an epidemic, and uncertainties in how human action will affect transmission. Realistic longer-term projections (spanning more than a few weeks) may, however, be possible under defined scenarios that specify the future state of critical epidemic drivers, with the additional benefit that such scenarios can be used to anticipate the comparative effect of control measures. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make 6-month ahead projections of the number of SARS-CoV-2 cases, hospitalizations and deaths. The SMH released nearly 1.8 million national and state-level projections between February 2021 and November 2022. SMH performance varied widely as a function of both scenario validity and model calibration. Scenario assumptions were periodically invalidated by the arrival of unanticipated SARS-CoV-2 variants, but SMH still provided projections on average 22 weeks before changes in assumptions (such as virus transmissibility) invalidated scenarios and their corresponding projections. During these periods, before emergence of a novel variant, a linear opinion pool ensemble of contributed models was consistently more reliable than any single model, and projection interval coverage was near target levels for the most plausible scenarios (e.g., 79% coverage for 95% projection interval). SMH projections were used operationally to guide planning and policy at different stages of the pandemic, illustrating the value of the hub approach for long-term scenario projections.


Table of Contents
. Projections evaluated Figure S1 Table S1-Table S2 S2. Evaluating scenario plausibility Figure S2- Figure Table S1: Summary of individual models that submitted projections in the first sixteen rounds of the U.S. COVID-19 Scenario Modeling Hub (SMH).

CU-AGE-ST Columbia University
The CU-AGE-ST uses a combination of two models for producing age-stratified, state level projections of SARS-COV-2 in the United States: a metapopulation, non-age-stratified model for a single strain and a state level compartmental model, population and age-stratified. Specifically, the inference step is performed with a metapopulation model that reproduces transmission within and between the 3,142 counties in the United States and assimilates daily county cases to estimate the county-level distribution of parameters and variables. Intercounty mobility is modeled using commuting patterns from U.S. Census Bureau, adjusted with Safegraph mobility observations from 2020. The inferred parameters and initial conditions are aggregated at the state level and combined with state-level data on population structure, age and state-specific seroprevalence estimates and published estimates of age-specific reporting rates, hospitalization rates, and death rates. The Centers for Disease Control and Prevention's (CDC's) Nationwide Commercial Laboratory Seroprevalence Survey estimates are used to assess the relative difference in age-specific susceptibility and age-specific incidence. Projections are produced with a separate compartmental model run in isolation for each of the 50 U.S. states and the District of Columbia (DC). This second model is stratified in 12 age and population-specific groups to reproduce different patterns of disease severity, different impact of non-pharmaceutical interventions (NPIs), and vaccine prioritization. The model imposes a seasonal forcing on the real time reproductive number.

JHUAPL-Bucky Johns Hopkins Applied Physics Laboratory
The JHUAPL-Bucky model is an age-stratified, county-level susceptible-exposed-infected-recovered (SEIR) model. The Bucky model estimates cases, hospitalizations, and deaths due to COVID-19 across all counties and territories in the United States.
Geographic spatial considerations are incorporated via public mobility data which accounts for inter-county mobility. Contact matrices derived from Prem et al. (1) are used to account for interactions between age groups within a given county/territory. In addition to static data sources related to the demographic distribution of individuals, dynamic input to the model consists primarily of incident case, hospitalization and death data from the two weeks preceding a simulation. These input data are smoothed/approximated with a generalized additive model.
Both static and dynamic input data sources are used to direct model parameter estimation methods. Parameters are selected via a joint optimization of all the priors via a Bayesian optimization procedure to maximize the coverage of our confidence intervals on the historical input data. Parameters (case reporting rates, doubling times, initial conditions, etc.) are estimated locally at the county or state level in order to account for differences in disease and transmission properties due to variation in population demographics and variant prevalence.
To characterize uncertainty associated with the output of a given simulation, two-thousand Monte Carlo runs are performed over multiplicative distributions of the individual parameter estimates. Predictive quantiles are computed from the outputs of these simulations.

JHU_IDD-CovidSP Johns Hopkins University
The Flexible Epidemic Modeling Pipeline (flepiMoP, formerly covidSP) models transmission of SARS-CoV-2 in the United States at the state level using a compartmental metapopulation structure, where U.S. states are connected through human mobility informed by commuting census data. The model compartments have varied over the course of the COVID-19 pandemic and between Scenario Modeling Hub rounds, depending on round specifications. In round 12, our model represented the population though 300 compartments per state, built from all the possible combinations of five disease stages (Susceptible, Exposed, Infectious, Recovered, and recovered with Waned immunity; SEIRW), five vaccination statuses (unvaccinated, vaccinated with 1 dose, vaccinated with 2 doses or boosted, vaccinated with waned vaccine-derived immunity, and unvaccinated with prior infection), four variants (Wild, Alpha, Delta, Omicron), and three age-strata (below 18, 18 to 64, above 65). Transmission is simulated from January 1st, 2020, and the early infections of each variant are seeded into the different states according to their first observations. The transitions between the compartments are simulated by integrating the governing coupled ordinary differential equations. Transition rates are modified by state-and timespecific non-pharmaceutical interventions, state-specific seasonality of SARS-CoV-2 transmission, and local variation in overall transmissibility by state. The incidence of infection from the dynamical model is passed to an observation model that computes observed cases, hospitalizations, and deaths, where the case detection probability varies in time and space, and the infection fatality ratio (IFR) varies by state, age group, vaccination status, and variant. While many parameters are derived from the literature (e.g., vaccine-induced protection against infection or death), other parameters are fitted (infection to case ratio, impact of non-pharmaceutical interventions, seasonality, and local transmissibility). We calibrate the model to CSSE data for death and cases in each state along with the variant proportions from CoVariants.org (GISAID) through our custom inference algorithm. This algorithm enables Bayesian inference on large-scale dynamic models through multi-chain Markov Chain Monte Carlo (MCMC), leveraging parallel computing resources with many short chains to handle large-scale epidemic dynamics and high-dimensional parameter space.

Karlen-pypm University of Victoria
Models built with the python population modeling framework (www.pypm.ca) use finite time difference equations to model a homogeneous population that reproduces the cases, hospitalizations, and deaths time-series data for the jurisdictions under study. The infection rate is proportional to the product of transmission rate, susceptible fraction, and number of circulating contagious individuals. The transmission rate is piecewise constant, adjusted to match case rates, typically constant over a period of 2 months. Adjustable time delays for partial propagations from infected to symptomatic to reported cases to hospitalization and to death are fit to match data. Reporting fraction is adjusted to match seroprevalence data. Multiple infection cycles are included to model variants, using a common susceptible population. An additional susceptible population is available to variants that partially escape from immunity in the population. Selection coefficients estimated from genomic data are used to set transmission rates for new variants as they emerge. Vaccination implemented with a fraction of those vaccinated reducing the susceptible population. Waning of vaccine and natural immunity are included with adjustable time delays to repopulate the susceptible population. The models can produce time series of expectation values and time series of simulated stochastic data. The former is used to produce point estimates for future rates and the latter to produce intervals.

MOBS_NEU-GLEAM_COVID Northeastern University
The model is a U.S. specific metapopulation compartmental model that refines our Global and Epidemic modeling approach. The GLEAM model is a stochastic, spatial, age-structured metapopulation model. Previously this model was used to characterize the early stage of the COVID-19 epidemic (2). Each subpopulation is defined as the catchment area around major transportation hubs. The airline transportation data encompass daily travel data in the origin-destination format from the Official Aviation Guide database50 reflecting actual traffic changes that occurred during the pandemic. Ground mobility and commuting flows are derived from the analysis and modeling of data collected from the statistics offices of 30 countries on 5 continents. The model accounts for travel restrictions and government-issued policies. Furthermore, the model accounts for the reduction of internal, countrywide mobility and changes in contact patterns in each country and state. The U.S. local epidemic and mobility model (LEAM) is enhanced to consider as a single subpopulation each one of the 3,142 counties (or its statistical equivalent) for each one of the 50 U.S. states. Population size and county-specific age distributions reflect Census' annual resident population estimates for year 2019. Commuting flows between counties are obtained from the 2011-2015 5-Year ACS Commuting Flows survey and properly adjusted to account for differences in population totals since the creation of the dataset. Contact matrices, age-specific traveling probabilities, and air traffic flows are in common with GLEAM and they are properly mapped at LEAM-US county-level resolution. Google's COVID-19 Community Mobility Reports data collected at the county-level resolution are used to model mobility and the effects of NPIs on individual behavior. The transmission dynamics take place within each subpopulation and assume a classic compartmentalization scheme for disease progression similar to those used in several largescale models of SARS-CoV-2 transmission. Each individual, at any given point in time, is assigned to a compartment corresponding to their particular disease-related state. This state also controls the individual's ability to travel. Individuals transition between compartments through stochastic chain binomial processes. The compartmental structure includes vaccination status and has been extended during the different projections rounds to accommodate new variants through an explicit multi-strain compartmental structure The model calibration is performed for each state through an Approximate Bayesian computation rejection algorithm using as evidence the weekly ground truth data for deaths, hospitalizations and cases.

NCSU-COVSIM North Carolina State University
The COVSIM team uses a stochastic agent-based simulation to model disease spread in North Carolina. Agents are assigned age, race/ethnicity, and high risk health condition attributes to represent the underlying population. Agents have time-varying interactions in households, peer groups (school or workplace), and the community which serve as the driver for disease spread. We use a force of infection model to generate the time until the next infection occurs and select agents to be infected. Once an agent is selected they progress through an extended set of SEIR disease states, which is dependent on the agent's attributes, behaviors, and variants. We model masks, vaccines and booster uptake. We include waning immunity from recovered agents back to the susceptible state. Our model is manually calibrated using disease parameters from literature tailored to North Carolina.

NotreDame-FRED Notre Dame University
The Notre Dame team uses an agent-based model based on a modified COVID-19 version of the Framework for Reproducing Epidemic Dynamics (FRED). The model simulates daily interactions among agents in specific locations, such as households, schools, workplaces, or neighborhood areas. The synthetic population is representative of the demographic and geographic characteristics of the simulated States. An agent's health is represented as one of the following states, Susceptible -Exposed -Infectious -Recovered. The model includes circulating variants of SARS-CoV-2 with different characteristics of transmissibility and immunity scape. Recovered individuals are only protected against reinfections with the same variant, and partially protected against reinfection with a different variant for 8 months on average. NPIs are included in the model. Vaccination is included in the model with a prioritization strategy based on age. Three components of the vaccine are modeled, protection against infection, disease given infection, and protection against severe disease given symptoms. The model is calibrated using data on deaths and dominance of circulating variants.

University of Florida
Stochastic, discrete-time agent-based model (ABM) of SARS-CoV-2 transmission in the state of Florida derived from our existing ABM for dengue transmission (3,4). The model synthetic population includes 20.6 million people and 11.2 million locations (i.e., households, workplaces, schools, long-term care facilities, and hospitals). Transmission occurs via a daily cycle of location-based interactions, (e.g., within household, employee-customer interactions, or between households via social interactions). Individual infections may progress through SEIRD states (with multiple levels of severity, and hospitalization), using existing literature to parameterize infectious outcome probabilities and disease state durations. Probability of infection is modified by location-type-specific infection hazards, infecting viral strain, seasonality, and time-varying personal-protective behaviors. Disease severity depends on the age and health status (i.e. has comorbidity or not) of the agent. A time-varying reporting model serves to replicate empirical dynamic detection and reporting processes. The model represents vaccination through the first 3 doses using a generalized mRNA vaccine and delivers doses in an age-structured way that reflects empirical vaccine delivery in Florida.

University of North Carolina at Charlotte
The UNCC team builds a relatively simple statistical model to retrospectively fit COVID-19 spread. The underlying mechanism is an SEIR-type compartment model and we focus on projecting the cumulative case numbers. In the first few rounds (up to round 9), we fit an exponential model for the cumulative case for each state as well as at nation-level. Then we project the model to the scenario-specific time periods (e.g., 12, 26, or 52 weeks horizon). Hospitalization and death are modeled as a binomial outcome of cases with lags identified from literature. Starting from round 10, we have applied a multivariate long short term memory (LSMT) model to address both long-term and short-term dynamics of the COVID-19 dynamics. Multivariate LSMT also overcomes potential issues of arbitrarily assigned lags between case and hospitalization and death.

University of Southern California
We proposed the SIkJalpha model as a discrete-time compartmental model. The central idea is that new infections are created through interactions between the currently susceptible population and previously infected population with a rate depending on the time since infection. This is discretized into k bins of size J, i.e., past k periods, each of J units of time, determine k types of infected populations with k different infection rates. The learning of the parameters is done through a weighted linear regression where the weight of old observation decays as a power of alpha. The approach has evolved with time, depending on dataset availability, variables and decisions under consideration that define the scenarios, and new factors that we believed to have a significant impact. True infections were estimated using seroprevalence data until Round 13, after which wastewater data were incorporated. We used a sigmoid curve to fit and extrapolate vaccine adoption until Round 3, and a contagion model after Round 3. The model supported as many vaccine rounds as given in the United States. In Round 14, the model was updated to support an arbitrary number of vaccines. We assumed the vaccine and natural immunity model to be "all-or-nothing" until Round 7, after which a waning immunity model was used. Two variants were supported starting in Round 3 until Round 5, after which an arbitrary number of variants were allowed. We disaggregated the new infection time-series into multiple time-series variants using prevalence estimates. Each of the time-series is fitted competing for a common susceptible population to estimate variant-specific transmission rates. Since Round 10, the model supported immune escape variants as well. Additionally, we expanded the model to track all possible vaccines and natural immunity states to estimate the level of susceptibility in the population. The model supports arbitrary age structure.

UTA-ImmunoSEIRS
University of Texas at Austin UTA-ImmunoSEIRS team uses an age-structured COVID-19 SEIRS compartment model that tracks changes in the level of protection acquired from past infection and vaccination. They describe the changes in population-wide immunity resulting from three sources: Delta infections, Omicron infections, and vaccination. The level of each source of protection is explicitly modeled through a state variable. Natural infections increase the infection-acquired protection variables and primary and booster vaccines increase the vaccine-acquired protection variable. The levels of immunity wane at different speeds that are based on published estimates. The variables are used to reduce disease susceptibility and severity by inhibiting infections, symptomatic disease, hospitalizations, and deaths. The efficacy of each form of immunity depends on the relative prevalence of the circulating variants.

UVA-adaptive
University of Virginia UVA-adaptive (as of round 12) is a discrete time SEIR based model with explicit tracking of multiple tiers of vaccine induced immunity, and the latest variant of infection. The overall infection curve produced by the model was converted to reported cases using dynamic case ascertainment and calibrated to match observed ground truth at the state level. Case ascertainment over time was informed by CDC seroprevalence surveys (until Feb 2022), and then coarsely adjusted based on wastewater surveillance and rates of at-home testing. Growth of variants were obtained either from scenario specification (via seeding), or enforced through a prevalence curve based on the growth advantage. Past and ongoing vaccinations are obtained at the state level and appropriately assigned to eligible individuals according to immune stratification. Vaccine efficacy against infection/disease are assumed to be the same, and overall coverage (when relevant) are obtained from scenario specifications. Hospitalizations and death outcomes are obtained by using an age-stratified adjustment over three age groups (0-18, 19-64 and 65+) with scaling factors to match the respective ground truths. Uncertainty bounds are obtained by using an experiment design over other model parameters such as infectious/incubation periods, reporting delay and bounds on ascertainment rates.

University of Virginia
The UVA-EpiHiper model is an agent-based, individual level networked model. It computes stochastic transmissions of a disease in a synthetic contact network between individuals and stochastic state transitions within each individual following a disease model. Our disease model is an SEIR model expanded with asymptomatic, vaccinated, hospitalized, ventilated, and deceased states, and stratified by age group. Over the rounds, we have extended our disease model to represent multiple variants, waning immunity, and immune escape. The immune waning is modeled as transition from states with natural/vaccinal immunity to a partially susceptible state, and the time to transition is sampled independently for each individual.
Our model is initialized with (i) county level data of prior infections (part of which have waned immunity) and recent confirmed case counts and (ii) state level data of prior vaccinations (part of which have waned immunity). Prior infections are derived from confirmed cases using age stratified ascertainment rates. We have modeled the following NPIs: (i) generic social distancing, the compliance to which changes over rounds; (ii) school closure during winter and summer breaks, and mask mandate in schools; (iii) voluntary home isolation of symptomatic people. Vaccines are applied to eligible individuals according to state level vaccine administration data and projection of future coverage as specified by scenarios. Our model is calibrated at state level targeting the estimated effective reproduction number at the beginning of the projection period.
We run simulations for each state and combine the output to get results of the whole United States. The simulations produce daily infections, hospitalizations, and deaths; and each simulation runs for multiple replicates. We aggregate daily data to get weekly data and compute quantiles for each target from the multiple replicates.      Scenario assumptions (green and purple dots) stipulated a median time in weeks to loss of immunity (x-axis) and a level at which protection against symptomatic disease stabilized after immunity loss (y-axis). Scenario assumptions did not distinguish between hybrid immunity (immunity from natural infection and vaccination) and natural immunity, but based on these graphs, the optimistic assumption was most realistic in both situations. Table S3: Scenarios by round. For each round, the major assumptions of each scenario (A-D) and the observed values corresponding to each major assumption are specified. Scenarios retrospectively determined to be "most realistic" are denoted with an asterisk (*). If a single most likely scenario could not be determined (here, due to insufficient data on non-pharmaceutical interventions (NPIs) for scenarios in Rounds 3-5), multiple most likely scenarios were considered. Projection period is represented weekly, where weeks are defined on Saturdays. For a full list of scenario specifications by round, see https://github.com/midas-network/covid19-scenario-modelinghub/tree/master/previous-rounds.

Round 1 Jan 9 to Jul 3, 2021 (alpha variant truncation on Apr 3, 2021)
Scenario A: optimistic* • Social distancing: NPIs continue for six weeks from their start date, interventions step down from baseline to lowest levels seen since September 2020 over two one-month steps • Vaccination: 50 million total doses distributed per month (~292 million total first doses distributed, ~140 million total doses distributed before truncation); 95% vaccine efficacy after two doses, 50% after one dose Scenario B: moderate • Social distancing: NPIs continue for three weeks from their start date, interventions step down from baseline to lowest levels seen since May 2020 over two one-month steps • Vaccination: 25 million total doses distributed in January 2021, 50 million for all other months (~273 million first doses distributed, ~122 million total doses distributed before truncation); 70% vaccine efficacy after two doses, 20% after one dose Scenario C: fatigue • Social distancing: NPIs continue for three weeks from their start date, interventions step down from baseline to 5% below lowest levels seen since May 2020 over two one-month steps • Vaccination: 25 million total doses distributed in January 2021, 50 million for all other months, no more than 50% of any priority group accepts the vaccine (~273 million first doses distributed, ~122 million total doses distributed before truncation); 95% vaccine efficacy after two doses, 50% after one dose Scenario D: counterfactual • Social distancing: NPIs continue for three weeks from their start date, interventions step down from baseline to lowest levels seen since May. 2020 over two one-month steps • Vaccination: no vaccines distributed (0 total doses distributed)

Observed
• Social distancing: despite the availability on mobility and policy data, the specificity of these data does not match that of our NPI scenarios; therefore we retain both moderate and low NPI scenarios as likely • Vaccination: ~361 million total doses distributed; ~186 million first doses distributed before truncation; 50% coverage achieved in 65+ by May 5, 2021; 80-90% vaccine efficacy against disease after two doses (5) Scenario C: fatigue, no variant • Social distancing: NPIs continue for three weeks from their start date, interventions step down from baseline to 5% below lowest levels seen since May 2020 over two one-month steps • Vaccination: monthly administration follows rates seen in January 2021; no more than 50% of any priority group accepts the vaccine (~124 million first doses administered, ~104 million first doses administered before truncation); 95% vaccine efficacy after two doses, 50% after one dose • Variant: no new variant Scenario D: fatigue, variant • Social distancing: NPIs continue for three weeks from their start date, interventions step down from baseline to 5% below lowest levels seen since May 2020 over two one-month steps • Vaccination: monthly administration follows rates seen in January 2021; no more than 50% of any priority group accepts the vaccine (~114 million first doses administered, ~96 million first doses administered before truncation); 95% vaccine efficacy after two doses, 50% after one dose at 50% of the effectiveness observed in February 2021 • Vaccination: 35 million first doses administered per month, no more than 90% of any population group receives the vaccine (~212 million first doses administered, ~130 million first doses administered before truncation); 95% vaccine efficacy against disease after two doses, 90% after one dose Scenario B: high vaccination, low NPI* • NPIs: NPIs decline gradually over a period of 5 months ending in August 2021 at 20% of the effectiveness observed in February 2021 • Vaccination: 35 million first doses administered per month, no more than 90% of any population group receives the vaccine (~212 million first doses administered, ~130 million first doses administered before truncation); 95% vaccine efficacy against disease after two doses, 90% after one dose Scenario C: low vaccination, moderate NPI • NPIs: NPIs decline gradually over a period of 5 months ending in August 2021 at 50% of the effectiveness observed in February 2021 • Vaccination: 20 million first doses administered per month, no more than 50% of any population group receives the vaccine (~108 million total doses administered, ~75 million total doses administered before truncation); 75% vaccine efficacy against disease after two doses, 50% after one dose Scenario D: low vaccination, low NPI • NPIs: NPIs decline gradually over a period of 5 months ending in August 2021 at 20% of the effectiveness observed in February 2021 • Vaccination: 20 million first doses administered per month, no more than 50% of any population group receives the vaccine (~108 million total doses administered, ~75 million total doses administered before truncation); 75% vaccine efficacy against disease after two doses, 50% after one dose Observed • NPIs: despite the availability on mobility and policy data, the specificity of these data does not match that of our NPI scenarios; therefore we retain both moderate and low NPI scenarios as likely • Vaccination: ~150 million first doses administered, ~122 million total first doses administered before truncation (excluding J&J, which was not considered in scenarios); 50% coverage is reached nationally on May 13, 2021, 90% coverage is never reached; 80-90% vaccine efficacy against disease after two doses (5)

Observed
• NPIs: despite the availability on mobility and policy data, the specificity of these data does not match that of our NPI scenarios; therefore we retain both moderate and low NPI scenarios as likely • Vaccination: national saturation at 78% of vaccine-eligible population by October 31, 2021 (note: because this round is truncated early in the projection period, we also consider the cumulative coverage gains during the projection period, 0.2% per day; if we extend this for the duration of the projection period, we have an expected coverage of 88%)                                       Figure S45: 95% prediction interval coverage, normalized weighted interval score, precision, and recall by round and target. Results are calculated for plausible scenario-weeks. Color scales show rank of each round by target and metric: ideal coverage ranked by absolute distance from 95% coverage, WIS ranked smallest to largest, precision and recall: red color scale shows, and blue color scale shows distance from 0. The number of plausible weeks evaluated is listed below each round (x-axis). For these results, precision and recall are averaged across all three classes (increasing, flat, and decreasing).               Projections are from the same model with two different assumptions about the specifics of how immunity wanes (waning times are assumed to be exponentially distributed for projections in pink, and assumed to be gamma distributed for projections in blue), despite both having the same scenario specified average duration and final protection levels. Line shows median projection, and ribbon shows 95% prediction interval.

S5. Additional performance results
Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.