Abstract
Importance Systematic characterization of the protective effect of vaccinations across time and at-risk populations is needed to inform public health guidelines and personalized interventions.
Objective To evaluate the vaccine effectiveness (VE) over time and determine differences across demographic and clinical risk factors of COVID-19.
Design, Setting, and Participants This test negative design consisted of adult patients who were tested or diagnosed for COVID-19 at Michigan Medicine in 2021. Variables extracted from Electronic Health Records included vaccination status, age, gender, race/ethnicity, comorbidities, body mass index, residential-level socioeconomic characteristics, past COVID-19 infection, being immunosuppressed, and health care worker status.
Exposure The primary exposure was vaccination status and was categorized into fully vaccinated with and without booster, partially vaccinated, or unvaccinated.
Main Outcomes and Measures The main outcomes were infection with COVID-19 (positive test or diagnosis) and having severe COVID-19, i.e., either being hospitalized or deceased. Based on these, VE was calculated by quarter, vaccine, and patient characteristics.
Results Of 170,487 COVID-19 positive adult patients, 78,002 (45.8%) were unvaccinated, and 92,485 (54.2%) were vaccinated, among which 74,060 (80.1%) were fully vaccinated. COVID-19 positivity and severity rates were substantially higher among unvaccinated (12.1% and 1.4%, respectively) compared to fully vaccinated individuals (4.7% and 0.4%, respectively). Among 7,187 individuals with a booster, only 18 (0.3%) had a severe outcome. The covariate-adjusted VE against an infection was 62.1% (95%CI 60.3–63.8%), being highest in the Q2 of 2021 (90.9% [89.5–92.1%]), lowest in Q3 (60.1% [55.9–64.0%]), and rebounding in Q4 to 68.8% [66.3– 71.1%]). Similarly, VE against severe COVID-19 overall was 73.7% (69.6–77.3%) and remained high throughout 2021: 87.4% (58.1–96.3%), 92.2% (88.3–94.8%), 74.4% (64.8–81.5%) and 83.0% (78.8–86.4%), respectively. Data on fully vaccinated individuals from Q4 indicated additional protection against infection with an additional booster dose (VE-Susceptibility: 64.0% [61.1–66.7%] vs. 87.3% [85.0–89.2%]) and severe outcomes (VE-Severity: 78.8% [73.5–83.0%] vs. 94.0% [89.5–96.6%]). Comparisons between Pfizer-BioNTech and Moderna vaccines indicated similar protection against susceptibility (82.9% [80.7–84.9%] versus 88.1% [85.5– 90.2%]) and severity (87.1% [80.3–91.6%]) vs. (84.9% [76.2–90.5%]) after controlling for vaccination timing and other factors. There was no significant effect modification by all the factors we examined.
Conclusions and Relevance Our findings suggest that COVID-19 vaccines offered high protection against infection and severe COVID-19, and showed decreasing effectiveness over time and improved protection with a booster.
Question How do the rates of COVID-19 outcomes (infections or mild/severe disease) compare across vaccination status and quarters of 2021, after adjusting for confounders?
Findings In this cohort of 170,487 adult patients tested for or diagnosed with COVID-19 during 2021, both COVID-19 positivity and severity rates were substantially higher in unvaccinated compared to fully vaccinated individuals. Vaccine effectiveness estimation was adjusted for covariates potentially related to both being vaccinated and COVID-19 outcomes; this also allowed us to determine if effectiveness differed across patient subgroups. The estimated vaccine effectiveness across the four quarters of 2021 was 62.1% against infection and was 73.7% against severe COVID-19 (defined as hospitalization, ICU admission, or death). There was no significant effect modification by all the factors we examined.
Meaning These findings suggest COVID-19 vaccines had relatively high protection against infection and severe COVID-19 during 2021 for those who received two doses of an mRNA vaccine (Moderna or Pfizer-BioNTech) or one dose of the Janssen vaccine, of which the effectiveness decreased over time and improved with a booster.
Introduction
Three Coronavirus Disease 2019 (COVID-19) vaccines were developed, assessed for efficacy against symptomatic COVID-19 disease in placebo-controlled trials, and approved under emergency use authorization in the United States by February 2021: mRNA-1273 (Moderna), BNT162b2 (Pfizer-BioNTech), and Ad26.COV2.S (Johnson & Johnson-Janssen). As of January 26, 2022, 536 million doses have been administered, and 210 million people are fully vaccinated, meaning they received at least one dose of Janssen or two doses of Pfizer-BioNTech or Moderna vaccine, in the United States1. In light of new SARS-CoV-2 variants, observational studies can provide insight on vaccine effectiveness (VE) with respect to severity of disease and measures of effectiveness over time as a proxy for protection against different variants.
Other studies have examined the effectiveness of the vaccines in real world settings over shorter and earlier periods of time and concluded the VE is relatively high against positivity and disease severity such as hospitalization and death2-9. While these studies support evidence of robust protection against variants with modest waning of VE through the third quarter of 2021, the FDA approved booster shots for continued protection of the vaccinated population over time. Still, the time-varying protection of vaccination over the course of 2021 and effectiveness among different vaccine brands remain unclear. More data from healthcare centers is needed to track hospitalizations and deaths across vaccination status. It is important to adjust for covariates that may be related to both getting vaccinated and severe COVID-19 outcomes as potential confounders in real-world studies10. Electronic health records (EHRs) are helpful for this type of research as they allow for such big data and covariate adjustment.
The objective of this work is to evaluate VE and determine if any patient characteristics show effect modification of VE using EHR from the University of Michigan Health System, in short Michigan Medicine (MM), a nationally ranked healthcare center and one of the largest health care complexes in the state of Michigan. Specifically, this time-stratified retrospective cohort study has two goals: (i) to investigate the COVID-19 VE against test positivity and severe COVID-19 outcomes across 2021, and (ii) to examine VE stratified by the two most common vaccines Pfizer-BioNTech and Moderna, and by sociodemographic and clinical characteristics that are associated with COVID-19 outcomes.
Methods
Study cohort and COVID-19 vaccination
Our study sample using the test negative design consisted of 170,487 adult patients who were tested or diagnosed for COVID-19 at MM during the four quarters of 2021, including 14,059 tested positive / diagnosed and 156,428 tested negative. Tested patients received a total 963,726 tests of the six different types: an in-house polymerase chain reaction (PCR) test (336,631 tests (34.9%)), a point of care PCR test (58,644 tests (6.1%)), a commercial PCR test (Viracor; 467 tests (0.048%); Quest 37 tests (0.004%)), COVID-19 nasopharynx or oropharynx PCR tests deployed by the Michigan Department of Health and Human Services (71 tests (0.007%)), and a small fraction of ribonucleic acid (RNA) tests (seven test (0.001%)); 567,869 test results (58.9%) were from patients transferred, tested elsewhere, or had no information on the test type.
Among all patients, 78,002 had no documented vaccination (“unvaccinated/unknown”) and 92,485 were vaccinated. Among those vaccinated, 74,060 were fully vaccinated according to FDA’s vaccination guideline, meaning they received either two doses of Moderna or Pfizer-BioNTech, or one dose of Janssen vaccine, and least 21 days had passed since completion of the series. The remaining 18,425 vaccinated patients either had not completed their vaccination series or did not adhere to the guidelines and received a mixed sequence of the above-mentioned vaccines (Table 1, Table S1). 254 individuals who received unspecified or other brand vaccines (AstraZeneca, Sinopharm, Sinovac, or Novavac) were not included in the study sample. Being boosted was defined as receiving any additional vaccination any time after the original full vaccination schedule.
COVID-19 outcomes
In this study, we focused on two COVID-19 associated outcomes: susceptibility, i.e., tested positive / diagnosed for COVID-19, and severity, i.e., having severe COVID-19 defined as either being hospitalized, admitted to an intensive care unit (ICU) within one month after a positive COVID-19 test / diagnosis, or died within two months after a positive COVID-19 test / diagnosis (Table 2). The summary of these outcomes is listed in Table 2, stratified by vaccination status.
Definition of demographics, socioeconomic status, and other covariates
We extracted the EHR data from patients with COVID-19 diagnoses or test results for COVID-19 (positive or negative) at MM, including age, self-reported gender, self-reported race/ethnicity, body mass index (BMI; calculated as weight in kilograms divided by height in meters squared), Neighborhood Socioeconomic Disadvantage Index (NDI) without proportion of Black (coded as quartiles, with larger quartiles representing more disadvantaged communities), and population density measured in persons per square mile (coded as quartiles). The Elixhauser comorbidity score developed by the Agency for Healthcare Research and Quality (AHRQ) was calculated to comprehensively characterize patients’ pre-existing comorbidity conditions using ICD9 and ICD10 codes and the R package “comorbidity” 11. Past COVID-19 infection was defined as testing positive at least 21 days before the first test of a quarter (if a person remained negative in that quarter), or 21 days before the first positive test of a quarter (if tested positive at least once in that quarter). Being immunosuppressed was defined for patients who had any immunosuppressants listed in their medication or prescription records within one year before a corresponding test (Calcineurin inhibitors, Interleukin inhibitors, TNF-alfa inhibitors, or any selective or other immunosuppressants)12. In addition, health care worker status was defined based on documented participation in a Health Care Worker Screen or a SARS-CoV-2 PCR test order for Health Care Workers. We report the variables’ missingness by vaccination status in Table S2. Missing data were handled by using a complete case analysis.
Statistical analysis
We fit Firth’s bias-corrected logistic regression for each COVID-19 related outcome YCOVID, considering several sets of covariates: where is the indicator of vaccination status, either with or without a booster, i.e. for them models comparing fully vaccinated individuals, and for the models comparing boosted individuals, and Covariates denotes the vector of covariates adjusted. Five nested sets of Covariates were evaluated, denoted as adjustment 1-5 (defined in Table S3). We reported the final model adjusted for age, gender, race/ethnicity, Elixhauser score, NDI without proportion of Black, population density, past COVID-19 infection, and health care worker status (adjustment 5).
We estimated the vaccine efficacy (VE) through the following formula 13: where Incidence Rate Ratio, the ratio of incidence between the vaccinated group and the unvaccinated group, can be estimated using the Michigan 12-month COVID-19 incidence rate (IR) and the estimated odds ratio (OR) from above logistic regression through the following formula 14: We estimated an IR of 0.088 based on the cumulative case count until December 29, 2021 (1,710,325 cumulative cases; JHU COVID-19 Dashboard 15 and the resident population of Michigan (Census 2020: 10,077,331) 16.
In addition, we conducted a quarter-stratified analysis to examine the time-varying changes in vaccination effectiveness over 2021: Quarter 1 (Q1), January 1, 2021–March 31, 2021; Quarter 2 (Q2), April 1, 2021–June 30, 2021; Quarter 3 (Q3), July 1, 2021–September 30, 2021; Quarter 4 (Q4), October 1, 2021–December 31, 2021.
To assess the potentially different VE across different strata of risk factors (denoted as X) associated with COVID-19 outcomes, we further conducted interaction analysis by vaccine status using the following model: All analyses were performed in R statistical software version 4.1.2 (R Project for Statistical Computing). For each model, we reported VE along with the corresponding 95% Wald confidence intervals (CI) and P values. For the interaction analysis, we obtained the subgroup OR based on βvac + βInt and reported the subgroup VE along with the P values, Pint, by testing the difference of subgroup effects via the null hypothesis H0 : βint = 0.
Results
Study characteristics
In our cohort of 170,487 adults, 92,485 (54.2%) had at least one COVID-19 vaccination dose before their test. The remaining 78,002 individuals had no documented vaccination record in their EHR and thus were considered unvaccinated though they may include individuals who received their vaccination elsewhere. A total of 74,060 had a complete vaccination series (“fully vaccinated”) and 18,425 had an incomplete vaccination series or a mix of vaccines, e.g., Moderna – Pfizer – Moderna (Table 1, Table S1).
We observed that mean age, Elixhauser Score, and proportion of health care workers was lowest in unvaccinated and highest in the fully vaccinated groups. The proportion of individuals who received Primary Care at Michigan was higher in the partially and fully vaccinated groups compared to the unvaccinated/unknown subset (38.4% and 44.8% versus 31.9%). Among those fully vaccinated, the 7,187 boosted individuals were characterized by the highest mean Elixhauser Score (8.9 versus <= 5.9) and an enrichment of individuals who received immunosuppressants in the past (11.4% versus <= 5.4%) (Table 1).
Observed Outcomes
When assessing COVID-19 positivity in the overall cohort, we observed substantially higher rates in unvaccinated (12.1%) compared to partially vaccinated (6.0%) or fully vaccinated individuals (4.7%). The positivity rate among fully vaccinated individuals reached a valley with low community transmission in Q2 (Q1: 1.3% and Q2: 1.1%) and surged to 3.2% in Q3 and to 6.7% in Q4 (Table 2) indicating potential waning effectiveness or consequences of spreading of the more immune evasive SARS-CoV-2 B.1.617.2 (Delta) and the B.1.1.159 (Omicron) variants17-19.
The overall rate of severe COVID-19 (hospitalization and/or ICU stay or death) was highest in unvaccinated (1.4%) compared to partially (0.5%) and fully vaccinated individuals (0.4%). Across the four quarters of 2021, the severity rate among fully vaccinated individuals remained relatively low (Q1: <0.01%, Q2: 0.1%, Q3: 0.3%, and Q4: 0.6%) compared to the unvaccinated group (Q1: 0.6%, Q2: 1.3%, Q3: 1.0%, Q4: 3.1%) indicating a less strong waning effect of the vaccine-mediated protection against severe COVID-19. Only 18 severe cases (0.3%) were observed among the 7,187 fully vaccinated individuals who received a booster shot (Table 2).
Vaccine Effectiveness
To next estimate the VE of a full vaccination against an infection (VE-Susceptibility) and against severe COVID-19 (VE-Severity), we applied logistic regression models unadjusted and adjusted for relevant covariates (see Methods, Table S3). To reduce bias in the estimates, it is important to adjust for potential confounding factors that may be related to both getting vaccinated (our exposure) and outcome (COVID positivity and severity) which include age, gender, race/ethnicity, comorbidity score, residential-level socioeconomic characteristics, past COVID-19 Infection, and health care worker status. In Figure 1, we present VE estimates for the overall and the time-stratified cohort (Q1, Q2, Q3, and Q4). Since the VE estimates were reasonably consistent across adjustments, e.g., in the overall cohort VE-Susceptibility ranged between 59.4 and 62.1%, we will only discuss the estimates from the most conservative setting (Adjustment 5 which included all above mentioned covariates) in the following. Results from all models are included in the corresponding figures for the sake of completeness. VE-Susceptibility in Q1 was 82.0% (76.0–86.5%), increased in Q2 to 90.9% (89.5–92.1%), and was substantially reduced in Q3 (60.1% [55.9-64.0%]) and Q4 (68.8% [66.3–71.1%]), suggesting waning vaccine effectiveness over time and potentially improved protection due to the increasing number of individuals who received booster shots (Figure 1A). This pattern was also observed for VE-Severity estimates though they were generally higher and did not drop off as dramatically (Q1: 87.4% [58.1–96.3%], Q2: 92.2% [88.3–94.8%], Q3: 74.4% [64.8–81.5%], Q4: 83.0% [78.8–86.4%]; Figure 1B).
Considering the increase in vaccine booster shots around Q4, we conducted a stratified analysis by booster status. These results of VE in Q4 demonstrated additional protection of a booster against infection (VE-Susceptibility: no booster: 64.0% [61.1–66.7%] vs. booster: 87.3% [85.0–89.2%]) and severe outcomes (VE-Severity: no booster: 78.8% [73.5–83.0%] vs. booster: 94.0% [89.5–96.6%]; Figure 1C, Figure S1).
We also performed a sensitivity analysis using only data on individuals who received primary care at Michigan Medicine and thus were more likely have fully documented vaccination data. In this analysis, all estimates were substantially higher for VE-Susceptibility (Q1: 88.7% [81.5–93.1%], Q2: 93.8% [92.3–95.0%], Q3: 68.7% [63.7–73.0%], Q4: 72.3 [68.9–75.4%]; Figure S2A) and VE-Severity (Q1: not available / no severe case among fully vaccinated, Q2: 95.2% [91.1–97.4%], Q3: 79.4% [68.0–86.7%], Q4: 80.8% [73.8–85.9%]; Figure S2B) indicating that the VE estimates in the full cohort might be influenced by incomplete vaccination documentation in the EHR (Figure S2).
Comparing VE between Pfizer-BioNTech and Moderna
Since the majority of the 74,060 fully vaccinated individuals completed a vaccine series with either Pfizer-BioNTech (n = 45,168, 61.0%) or Moderna (n = 25,267, 34.1%), we next compared the effectiveness of the two vaccines. To reduce the differing impact of a waning effectiveness, i.e., the Pfizer-BioNTech vaccine was available earlier than the Moderna vaccine, we focus on the analysis which only included individuals who reached full vaccination status less than 3 months before their COVID-19 test or diagnosis (Figure 2).
In the overall cohort, we observed lower VE-Susceptibility for Pfizer-BioNTech compared to Moderna (82.9% [80.7–84.9%] versus 88.1% [85.5–90.2%]), a trend that was consistent across the three quarters though lowest in Q2 (89.6% [87.6–91.3%] versus 91.7% [89.3–93.5%]; Figure 2, Figure S3). Contrarily, VE-Severity was higher for Pfizer-BioNTech (87.1% [80.3– 91.6%]) compared to Moderna (84.9% [76.2–90.5%]) in the overall cohort, consistent in the first half of 2021 but reversed in the last half of 2021. Of note, the VE-Severity in Q4 was substantially higher for Moderna (93.4% [5.3–99.6%]) compared to Pfizer-BioNTech (68.3% [23.6–87.2%]), though sample sizes were small, resulting in extremely wide and overlapping confidence intervals (Figure S4).
In general, we observed overlapping confidence intervals for all pairwise comparisons, suggesting similar protection against susceptibility and severity, though larger studies are needed to tease out the vaccines’ performance.
Factors Affecting Vaccine Effectiveness
Finally, we explored if VE varied across demographic and clinical risk factors. Overall, we found that a full vaccination was moderately effective against an infection across all strata (57.1% <= VE-Susceptibility <= 65.2%) except for people who had a documented past COVID-19 infection (VE-Susceptibility Past COVID-19 Infection = 47.6% [24.5% – 63.9%]; PInt. = 0.065); Figure 3). These lower estimates are expected, because unvaccinated individuals who already acquired immunity through a previous COVID-19 infection likely do not benefit as much from a vaccination as people who were not previously infected. It is important to note that only 182 individuals were reported as being reinfected in the data, and only 48 of these were fully vaccinated.
For health care workers (HCW) we observed a higher effectiveness compared to others (VE-Positivity HCW: 70.4% [62.8–76.5%] versus VE-Susceptibility Non-HCW: 61.7% [59.9–63.9%]; PInt. = 0.031), potentially indicating a stricter adherence to the vaccination regiment or additional protective measures 20,21.
Additionally, we observed decreased protection against an infection in individuals living in neighborhoods with increasing NDI quartiles compared to the lowest NDI quartile indicating potential health disparities (PInt. <= 0.023).
When we next focus on VE-Severity, we also observed effective protection against severe COVID-19 across all strata (VE-Severity >= 66.6%), except for the estimated reduced effectiveness for individuals with a past COVID-19 infection (VE-Severity Past COVID-19 Infection 45.3% [0% - 77.1%]; PInt. = 0.133). Due to the lower counts of individuals with severe COVID-19, confidence intervals were wider (Figure 3). None of the observed interactions remained significant after correction for multiple testing (16 tests).
Discussion
This study of 170,487 adults tested/diagnosed for COVID-19 at MM provides additional evidence of high COVID-19 vaccine effectiveness against SARS-CoV-2 variants across 2021. In our study, fully vaccinated individuals were more likely to test negative and were less likely to have severe disease than unvaccinated individuals or those with unknown vaccination status. Qualitatively, we observed waning effects of VE against positivity and disease severity across the four quarters. However, as the number of individuals who received booster shots increased in the fourth quarter of 2021, both infection and severity rates were lowest among boosted individuals. The Q4 increase in VE indicates improved protection through a booster when facing potential waning vaccine effectiveness and virus variants. The availability of potential confounding patient characteristics allowed us to evaluate potential modification of VE by patient subgroups. Further, our analysis compared the effectiveness of the first two FDA-approved mRNA vaccines. When including all study subjects, the Pfizer vaccine seemed to offer more protection against severe disease, while Moderna seemed to protect better against contracting COVID-19. Because Pfizer and Moderna vaccines were approved at different dates, we stratified by time since vaccination. When restricting to subsets of individuals who received the vaccine within the past three months, VE against infection tended to be slightly higher for those who received the Moderna vaccine. VE against severe disease was moderately high for both vaccines. The interpretation of changing VE is difficult as it can be affected by lower protection against the Delta or Omicron variants, waning vaccine effectiveness, or other considerations 4.
Our results are largely consistent with previously published findings, though differences among studies make direct comparison challenging, e.g., some studies adjusted for potential confounders (age only, other patient characteristics, or time since vaccination) while others characterized the variant (either Alpha, Mu, or Delta) associated with each case via whole-genome sequencing. Our study spans the whole year of 2021 and thus likely includes data on multiple variants (including the Omicron variant) and also on booster doses that have become more prevalent. From other studies conducted in late 2020 to early 2021, VE estimates against hospitalization ranged around 87% (CI 55-100%)2 and 89% (CI 87-91%)3 against infection. Studies during mid-2021 reported VE of 79.8%7 and 79.9-98.4%4 against infection and 95.3%7, 86% (CI 82-88%)5, 80-97.5%4, and 80.0% (CI 76.8-82.7%)9 against hospitalization across different variants. A recent study that looked further at the waning VE from late 2020 through Q3 of 2021 found VE rates for mRNA vaccines to be the highest two weeks after a two-dose vaccination regimen (94.5%; CI 94.1-94.9%), lining up with our estimates of VE in Q2, and lowest at 66.6% (CI 65.2-67.8%) after seven months8, which corresponds with our VE estimates in Q3. Taken together, these results highlight the need for appropriate adjustment for patient covariates, timing of vaccination, booster status, and for replication across different regions and time periods to more fully understand VE and potential VE modifying factors.
A strength of the present study is the availability of detailed EHR data for study subjects. This allows us to adjust for potential confounding patient characteristics including socioeconomic and demographic characteristics and relevant medical conditions including previous COVID-19 testing results and receipt of a booster dose. Most often, the test negative design has been used to evaluate VE3,22,23. As such, a limitation of results from this study design is the generalizability to individuals tested for COVID-19. Still, these studies provide insight about the long-term effectiveness and durability of vaccine protection23. A potential weakness is any missing information from the EHR, e.g., undocumented test results or vaccinations, though we believe categorizing individuals into an unknown/unvaccinated status tends to produce conservative VE estimates. Overall, these results should give confidence in the effectiveness of the vaccines and may encourage those who have not been vaccinated or not received a booster to do so.
Data Availability
The datasets generated during and/or analyzed during the current study are not publicly available due to MM privacy regulations.