Estimating area-level variation in SARS-CoV-2 infection fatality ratios

1 Introduction

During the course of an epidemic, accurate estimation of the infection fatality ratio (“IFR”) – defined here as the number of deaths caused by an infection per 1000 infected people – is crucial for policy and planning purposes. IFRs may vary by age distribution of the population, distribution of infection across demographic groups, prevalence of underlying health conditions in the population, access to healthcare resources, and other factors^1,2. Therefore, a flexible framework for estimating IFRs in different geographic regions and among different populations is important.

Accurately ascertaining both the number of deaths and the number of infections can be challenging³. While the number of deaths may be under- or over-reported in some settings⁴, even more concerning are potential inaccuracies in the ascertainment of the number of infections. While case mortality data may provide an upper bound for the IFR, the total number of infections is likely to be underestimated because infected people may not be tested or report illness. In addition, the availability and uptake of testing, as well as reliability of testing data, may also vary over time and geography. For example, during the COVID-19 pandemic, the number of SARS-CoV-2 nucleic acid amplification tests (NAATs) administered was consistently recorded at the state level across the U.S., though not at the county level⁵. Focusing on the COVID-19 pandemic in the U.S., here we develop an inferential framework for estimating area-level variation in testing efforts, number of SARS-CoV-2 infections, and SARS-CoV-2 IFRs. Although this framework was developed to address the need for countylevel SARS-CoV-2 IFR estimates in the U.S., it can readily be applied to other geographic settings and pandemics to estimate area-level variation in outcomes.

2 Methods

This analysis aims to develop estimators for the SARS-CoV-2 IFRs – defined here as the number of COVID-19 deaths per 1000 SARS-CoV-2 infections – within each U.S. county. Here we focus on data from March 1, 2020 - October 31, 2020, though this approach can be applied to any time period within the pandemic and to diseases other than COVID-19. There are two primary challenges with developing these estimators for COVID-19. First, the total number of SARS-CoV-2 infections is unknown because most asymptomatic individuals, but also many symptomatic individuals, fail to get tested or report illness^6,7,8. Second, although reliable data are available on the number of SARS-CoV-2 NAATs administered per state, reliable testing data are not available at the county level^5,9. Having data on testing is needed because it allows sampling effort to be factored in; without it, for instance, it is impossible to know whether observing one or a few cases in a county is indicative of a low SARS-CoV-2 infection ratio or just a limited testing effort. What follows is an inferential framework for estimating the testing effort, number of SARS-CoV-2 infections, and ultimately SARS-CoV-2 IFRs for each county.

2.2 Assumptions

To derive an estimator for the county-level IFR, assumptions about several quantities are necessary:

1. Odds ratios (ω): Although it is not necessary to assume actual values of ω, it is necessary to assume a geographic structure for them for each time point. Here we assume that at a given time point in the pandemic, the odds ratio of testing an infected vs an uninfected individual is the same for all counties, states, and the entire U.S. That is, ω_c = ω_s = ω_n. However, the odds ratios can change through time.

2. Global IFR (ϕ_n): To estimate geographic variation in IFR, it is necessary to assume an baseline IFR for the entire region under consideration. In our case, we assume that the IFR across the entire U.S. was five deaths per 1000 infections from March 1, 2020 to October 31, 2020, a widely agreed upon value¹⁰.

3. Observed quantities: we assume that the population (P) and numbers of COVID-19 deaths (D) and cases (C) in each county, state, and across the entire U.S. are known. Likewise, we assume that the number of tests in each state (T_s) and consequently across the whole U.S. (T_n) is known. However, we assume that the number of tests in each county is unknown (T_c). While the latter assumption may seem peculiar in light of the assumption that T_s is known, it reflects the situation that, in the U.S., the number of SARS-CoV-2 NAATs in each county was not consistently recorded during much of 2020, despite the fact that it was recorded in aggregate for each state.

4. Number of tests in each county and state (T_c and T_s): Because the number of NAATs in each county is unknown, it is necessary to estimate it. To do so, we assume that the number of tests administered T_cj in county j is related to the population size P_cj and the number of observed cases C_cj by , for some constants α, β, and γ, where ϵ_cj is distributed normally with variance zero. Thus, log T_cj is a linear function of P_cj and C_cj: logT_cj = α + β logP_cj + γ log C_cj + ϵ_cj. We make an analogous assumption for states, i.e., that , implying that log T_sj = α + β log P_sj + γ log C_sj + ϵ_sj. Empirically, these assumptions are strongly supported for states (Figure 1). Furthermore, it reasonable to suppose that for some non-negative constants κ and η, which are additionally upper-bounded by 1: T_ij = κP_ij and C_ij = ηT_ij, which is to say, that (a) the number of tests in state or county j is directly proportional to the population size of that state or county and (b) the number of observed cases in a state or county is directly proportional to the number of tests conducted there. Multiplying these two equations and simplifying yields:

• Download figure
• Open in new tab

Figure 1:

Relationship between the number of NAATs performed and the total number of COVID-19 cases and population size at the state level. Although the intercepts and slopes of these relationships varied over time, all are approximately linear when log-transformed. By applying linear models, fitted using state-level data, for the number of log tests at each time point as function of the log cases and log population size, we inferred the number of tests in each county at each time point. This inference was possible because, although in most counties the number of tests was not recorded, the number of COVID-19 cases and population size were available for all counties.

That is, the power function assumed above.

At a given time point within a specific region, it is reasonable to conceptualize the process by which people are tested with a noncentral hypergeometric model, wherein each test is administered to either a person having or not having COVID-19 with some bias toward administering tests to symptomatic people (e.g., because sick people are more likely to seek testing⁸; quantified by ω). For estimation, we utilize an approximation for the mean of the Wallenius’ noncentral hypergeometric distribution (Equation 16 in reference¹¹): While recognizing that this expression is a close approximation, for clarity, in the following we will refer to it as an equality.

3 Results

Applying the county-level estimators to the real, unsimulated COVID-19 data spanning March 1, 2020 to October 31, 2020 from across the U.S., we found that IFRs varied from 0 to 0.0273, with a mean of 0.0023 and a median of 0.0018 (interquartile range 0.0022). IFRs were the lowest in the Intermountain West, High Plains, and New England, but the highest in Arizona and Western New Mexico, the Upper Midwest, and scattered other regions of the U.S. The number of SARS-CoV-2 infections (not corrected for population) was highest in the Southwest and scattered other parts of the U.S., and lowest in Utah. Testing (also not corrected for population) was generally highest around large population centers. Similar trends were evident at the state level, with lower spatial resolution (Figure 2).

• Download figure
• Open in new tab

Figure 2:

Maps showing estimated and observed values of the number of deaths, cases, infection fatality ratios, number of infections, and number of tests in the U.S. over the period spanning March 1, 2020 to October 31, 2020. Estimated quantities are shown in color, while observed quantities are shown in gray. The number of tests was observed at the state level but unobserved at the county level, where it is estimated.

In our validation, we found that the mean of the county-level estimates of IFRs, number of SARS-CoV-2 infections, and tests had high accuracy (Figure 3; Pearson correlation coefficients for estimator means and true values 0.88, 0.95, and 0.74, respectively). Likewise, at the state level, the mean of the estimates for IFRs and number of infections had high accuracy and precision (Figure 4; Pearson correlation coefficients for estimator means and true values 0.95 and 0.99, respectively). Moreover, the estimators had coefficients of variation below 0.1 for almost all counties and states (Figure 5; for example, for IFR, 98.4%, 92.3%, and 11.6% of counties had coefficients of variation below 0.1, 0.05, and 0.01, respectively). In contrast, naive IFR estimates obtained simply by using the numbers of cases and case fatality ratio were biased by up to several orders of magnitude and had low precision (Figure 6; Pearson correlation coefficients for estimator means and true values were 0.21 and 0.51, respectively).

• Download figure
• Open in new tab

Figure 3:

Performance of county-level estimators. Maps in the first column show true (simulated) infection fatality ratios, numbers of infections, and tests in each county. All values were conditioned on observed numbers of deaths. Counties with zero deaths are omitted from the IFR estimation results because they have ratios of zero. The second column gives the means of the estimates over 100 simulated data sets for each county. The close match between true values and estimates indicates that the estimators have high accuracy. The accuracy is further illustrated by the graphs in the third column, which plot the true values in each county compared to the mean of the estimator. In each plot, counties (x-axis) are ordered by increasing value.

• Download figure
• Open in new tab

Figure 4:

Performance of state-level estimators. The maps and graphs are analogous to those in Figure 3, but are at the state level. Unlike at the county level, at the state level the the number of tests is observed rather than estimated.

• Download figure
• Open in new tab

Figure 5:

The estimators for the IFR, number of infections, and number of tests have high precision. At both the county and state levels, the estimators have low coefficients of variation, generally below 0.1 and often below 0.01.

• Download figure
• Open in new tab

Figure 6:

The estimators developed here greatly outperform uncorrected estimators of the IFR and number of infections, respectively. With simulated data where true values are known, at both the county and state levels, the case fatality ratio and number of cases overestimate and underestimate the IFR and number of infections, respectively, by up to several orders of magnitude. While this might seem expected, it shows that the estimators for developed here for the IFR and number of infections make substantial corrections.

4 Discussion

We present an inferential framework for estimating testing effort, number of SARS-CoV-2 infections, and ultimately SARS-CoV-2 IFRs in U.S. counties. The approach relies on a noncentral hypergeometric model that accounts for differential probability of positive tests (i.e. because sick individuals are more likely to seek testing than healthy individuals⁸) and performs well in validation testing.

Seroprevalence data have been widely used to estimate IFRs^12,13,14, and in limited settings, comprehensive case reporting and tracking exist¹⁵. We offer the proposed framework as an extension to such methods. Our estimates require an assumption about the baseline IFR (Assumption 2), which may be based on seroprevalence or detailed tracking data. However, such data are often limited geographically and may have non-representative samples or fail to account for potential delays from onset to seroconversion¹⁶. Therefore, we propose using data on testing effort, where reliably reported, to address area-level and temporal variation.

The number of excess deaths is another frequently used metric of the burden of mortality potentially related to the COVID-19 pandemic. Typically defined as the difference between the observed number of deaths and the expected number of deaths across a given time period¹⁷, it includes changes in mortality that may directly or indirectly be attributable to COVID-19 (such as such as from delayed care or behavioral health crises). While useful in some settings, excess death estimates are highly sensitive to the reference time period used and negative estimates often occur in younger age strata^18,19. More detailed area-level estimates of mortality directly attributable to an infection are often needed.

Similar to all models, our results are dependent on the quality of the data, which has limitations. Nonetheless, an advantage of our approach is that it allows investigators to rely on the best available data and estimate quantities where data may be lacking. Our estimates use data on NAAT (also known as molecular or PCR tests) and do not include data from antigen tests including home tests, which did not receive FDA approval until late 2020. The NAAT are the most reliable for detecting infections and were the most widely used during the study period.

Our approach relies on a number of assumptions that can be relaxed to allow for generalization. First, Assumption 1 – that the odds testing an infected vs an uninfected individual is the same for all counties, states, and the entire country – can be relaxed in numerous ways. While some geographic or temporal structure must be assumed to estimate the odds ratios, which are latent variables, this structure does not need to be geographically homogeneous. With additional covariates (e.g., data on demographics, health care access), it may also be possible to further infer specific aspects of the odds ratios. Second, regarding Assumption 2, while an overall IFR must be assumed, this value can in principle be any number between 0 and 1 dependent on the disease, time, location, and other factors. Moreover, when the aim is to rank IFR in subregions relative to each other instead of estimate their precise values, the assumed overall IFR may not matter. And third, an added degree of complexity specific to COVID-19 in the U.S. is imparted here because the number of NAATs in each state, but not county, are known. In a more typical scenario, when testing effort is recorded for each subregion, one could directly make estimates for a single class of subregions (e.g., counties) directly from a superregion (e.g., country). This would omit the need for our Assumption 4 and simplify Assumption 3. By virtue of its flexibility, our approach shows promise for application at different geographic scales and may be particularly useful in settings in which resources to carry out large, representative seroprevalence studies are not available.

Our approach can be flexibly adapted over time and space. This flexibility is important because the IFR is not biologically fixed. It varies with the distribution of age, health, and other qualities of those infected as well as the medical care they receive. Local estimates may be more accurate and useful for policy planning purposes than generalized or country-level estimates. For instance, population age structures do not address all of the heterogeneity between countries IFRs based on seroprevalence¹³. In conclusion, our approach can serve as an extension to other methods for estimation of pandemic burden to allow for greater precision around geographic and/or temporal population subgroups.