Influenza mortality as an indicator of the effectiveness of public health interventions and their impact on United States COVID mortality

Background

Since the outset of the COVID pandemic, non-pharmaceutical interventions (NPIs) to protect public health have come under heavy criticism for their impact on everything from the economy¹ to mental health^2–4 to education^5,6. Furthermore, almost every intervention has, at some point, been declared ineffective, including masking,^7,8 routine testing,⁹ school closures,¹⁰ and business “lockdowns”.^11,12

Meta-analyses of studies of the efficacy of individual NPIs have tended to find beneficial effects^13–15 with a few prominent exceptions.^7,8 Closures of businesses and schools, limits on social gathering, travel restrictions, social distancing rules, masking mandates, and other NPI’s act in concert to reduce the transmission of respiratory infections. Some protect the individual from the infection in the community, some protect the community from the infected individual, and some do both. Also, the effectiveness of NPI’s depends on compliance, which is difficult to quantify. How, then, do we evaluate the overall impact of these interventions on the transmission of COVID?

In an ideal natural experiment, we would have two isolated regions experiencing epidemic conditions that are identical in every way except for fully quantified and controlled differences in NPIs. Alternatively, we might have historical data for a particular disease and could examine changes in incidence and mortality after interventions were imposed. No such natural experiment occurred and, because COVID is a new human disease, we have no historical data. All of this makes the aggregate impact of NPI’s on COVID difficult to assess directly.

However, the impact of these NPIs was not limited to COVID. Interventions designed to stop one respiratory pathogen will stop others as well. Therefore, the extent to which these NPIs halted the spread of respiratory pathogens with similar patterns of airborne transmission may provide a surrogate for the efficacy of NPIs for COVID. By far the best characterized of these diseases is influenza.

The marked seasonal patterns of influenza incidence and mortality have been measured for decades. As a result, the expected influenza mortality and the variability in that mortality are well established. Multiple studies have noted the dramatic, global decline in influenza incidence and mortality ^16–18 during the COVID pandemic and suggested an association with NPIs. In the United States, influenza mortality rates for the first two complete flu seasons (2020-22) were 80% below historic rates,¹⁹ as can be seen in Figure 1. The sharpness of this decline and the fact that flu mortality rose back to pre-pandemic levels once precautions were lifted in 2022-23 makes it unlikely that the drop was due to any change in the prevalent virus or treatment options. Influenza vaccination rates did rise 7% above historical averages during the pandemic,²⁰ probably due to concurrent vaccination with COVID, but this cannot explain an 80% drop in influenza mortality.

• Download figure
• Open in new tab

Figure 1.

CDC estimates of deaths from influenza for past 13 flu seasons. Note that the CDC did not provide an estimate for the 2020-2021season because the mortality rates were too low for their estimation procedures, which seek to account for unreported cases, so the number provided is the actual count.

It appears that something changed during the pandemic that resulted in a dramatic drop in respiratory disease transmission. By far the most likely explanation of this is COVID NPI’s. That suggests that the extent to which influenza mortality decreased from expected levels represents a drop in respiratory disease transmission and may provide an indicator of the effectiveness of COVID NPI’s.

The current study explores the association between the influenza mortality reduction (IMR) and COVID mortality at the state level. Other factors considered in the analysis were COVID mortality during the first month of the pandemic and vaccination rates. Because the effectiveness of NPIs can be influenced by population density, it was also included in the model as an interaction term.

Methods

Weekly counts of influenza deaths for the period from 2016 through 2023 were abstracted from the CDC FluView System²¹ for each state. Average annual influenza mortality rates for each state were calculated for the pre-COVID period 2016 through 2019 and for the two flu seasons during the pandemic, 2020-21 and 2021-22. The decrease in average flu season mortality for each state during the pandemic as compared to average mortality rates prior to the pandemic were calculated for each state to determine the Influenza Transmission Control.

State specific, weekly COVID mortality data were obtained from the CDC COVID Data Tracker and used to calculate mortality rates for March of 2020 to determine initial COVID mortality rates.²² Vaccination rates at the end of 2023 were obtained from the same site with vaccination defined as receipt of two initial doses.²² Age adjusted COVID mortality was determined based on age specific COVID mortality rates from CDC through September, 2023, the last date for which they were available.²³ State age distributions and population density for 2020 was obtained from the United States Census Bureau.²⁴

Two multiple linear regression models were evaluated with state specific, age adjusted COVID mortality as the outcome variable. Model 1 included only four predictor variables, IMR, COVID mortality, vaccination rates, and population density. Model 2 added an interaction term for IMR and population density based on the assumption that the effect of NPI’s as represented by IMR would depend on population density. All statistical analyses were conducted using the STATA statistical package.

The resulting models were used to evaluate the counterfactual cases of non-intervention by setting the relevant variable to zero for each state, determining predicted deaths for each state, and summing the results. Since this effectively removed IMR as a variable, the interaction term also became meaningless, so Model 1 with no interaction term was used. Monte Carlo simulations with 10,000 repetitions were used to estimate the distribution of the aggregated mortality predictions and extract confidence intervals. The three cases considered involved setting IMR, vaccination rate, and both variables to zero.

Results

As listed in Table 1, state-level influenza mortality rates were an average of 78% lower during the two full flu seasons of the pandemic, 2020-21 and 2021-22, as compared to the three full flu seasons prior to the pandemic, 2016-17, 2017-18, and 2018-19. The decrease in mortality ranged from 49% (North Dakota) to 94% (Washington). This radical difference in influenza mortality during the pandemic at the state level is highly unlikely to reflect simple seasonal variation in the flu strain or vaccine effectiveness (p<<0.0001 by simple ANOVA).

COVID mortality had a strong negative correlation with IMR and vaccination rates (p<0.001). As shown in Figure 1, IMR explains almost a third of the variability in COVID mortality. IMR is also strongly correlated with vaccination rates (Table 2).

Multiple linear regression model results are provided in Table 3. Model 1, with only four predictor variables, IMR, COVID mortality, vaccination rates, and population density, predicts COVID mortality at the state level with an r² to 0.65. Introducing an interaction term for IMR and population density in Model 2 improves the adjusted model r² to 0.69. This interaction had a positive coefficient, suggesting the rate of reduction in COVID mortality associated with IMR was diminished in more densely populated states. Also, introducing the interaction to the model converted the direction of the effect of population density on COVID mortality from positive to negative. The close fit of the model to actual age-adjusted state COVID mortality rates can be seen in Figure 2.

• Download figure
• Open in new tab

Figure 2.

Actual age adjusted COVID mortality as compared to model estimates for US states. Using influenza mortality reduction as an indicator of their aggregate efficacy, this study shows that non-pharmaceutical interventions (NPI’s), vaccination rates, population density, and initial pandemic impact explains 69% of state specific COVID mortality through 2022 to assess their impact on COVID mortality.

With IMR set to zero, the model yields an estimated COVID mortality of 1.99 million (95% CI of (1.32 – 2.66 million) suggesting that NPIs saved 840,000 lives. If we set both IMR and vaccination to zero, the COVID mortality estimate rises to 3.39 million (95% CI of (2,93 – 3.86 million) suggesting that preventative measures saved 2,250,000 lives.

Discussion

The current study provides strong evidence that NPIs played a key role in limiting the impact of the pandemic preventing an estimated 830,000 COVID deaths respectively.

It is conceivable that a decrease in ascertainment rather than reduced transmission could contribute to a decline in influenza mortality. Some have even suggested that COVID deaths are actually influenza deaths.²⁵ Several observations allow us to dismiss these alternatives. First, failure to diagnose a fatal case of the flu correctly, even during the pandemic, seems unlikely given the well-established surveillance system and diagnostic tools for influenza. Second, the sharp drop in influenza incidence during the pandemic was observed in data from the Seattle Flu Study, which was an active surveillance program that demonstrated pandemic-related decreases for a broad range of respiratory infections.^26,27 All respiratory infections dropped sharply including influenza, respiratory syncytial virus, and non-COVID corona viruses. Finally, if flu deaths were being misdiagnosed as COVID, we would expect the reduction in influenza mortality to have a strong positive correlation with COVID mortality rates, not the strong negative correlation observed in these data.

It is notable that the regression coefficient for population density changes sign when interaction with IMR is included in the model. This may reflect the fact that population density is a two-edged sword with respect to COVID mortality, inferring a higher transmission risk but providing better access to life-saving medical care. Also of note is the negative association between the pre-intervention COVID mortality and total COVID mortality. This may reflect greater compliance with interventions in the states hardest hit at the outset of the pandemic.

One key advantage in using state IMR as a measure of NPI efficacy is that a region can serve as its own control. Comparing influenza mortality during the pandemic to historical mortality rates of influenza incidence and mortality with those that prevailed during periods when COVID NPIs largely eliminates the effect of time invariant confounders.

This model estimates that NPI’s alone, as indicated by IMR, prevented 840,000 COVID related deaths and that, with no public health interventions, COVID would have killed 3.3 million Americans. This is consistent with the controversial early estimates from the Imperial College of London,²⁸ although that relatively simple model assumed a far more rapid spread of the disease.

The ability of this relatively simple model to explain 69% of the variability in state COVID mortality provides compelling evidence that IMR is a useful indicator for the effectiveness of NPIs against COVID and that the factors included in the model were the primary drivers of COVID mortality. Although IMR appears to be an excellent indicator of the effect of NPIs, it does not provide any insight into exactly which interventions were effective. Understanding the contribution of various NPI’s to IMR will be critical to refining management strategies for future epidemics of respiratory infectious disease.