ABSTRACT
Background The CDC recently defined being “up-to-date” on COVID-19 vaccination as having received at least one dose of a COVID-19 bivalent vaccine. The purpose of this study was to compare the risk of COVID-19 among those “up-to-date” and “not up-to-date”.
Methods Employees of Cleveland Clinic in Ohio, USA, in employment when the COVID-19 bivalent vaccine first became available, and still employed when the XBB lineages became dominant, were included. Cumulative incidence of COVID-19 since the XBB lineages became dominant was compared across the ”up-to-date” and “not up-to-date” states, by treating COVID-19 bivalent vaccination as a time- dependent covariate whose value changed on receipt of the vaccine. Risk of COVID-19 by vaccination status was also evaluated using multivariable Cox proportional hazards regression adjusting for propensity to get tested for COVID-19, age, sex, most recent prior SARS-CoV-2 infection, and number of prior vaccine doses.
Results COVID-19 occurred in 1475 (3%) of 48 344 employees during the 100-day study period. The cumulative incidence of COVID-19 was lower in the “not up-to-date” than the “up-to-date” state. On multivariable analysis, being “up-to-date” was not associated with lower risk of COVID-19 (HR, 1.05; 95% C.I., 0.88-1.25; P-value, 0.58). Results were very similar when those 65 years and older were only considered “up-to-date” after 2 doses of the bivalent vaccine.
Conclusions Since the XBB lineages became dominant, adults “up-to-date” on COVID-19 vaccination by the CDC definition do not have a lower risk of COVID-19 than those “not up-to-date”, bringing into question the value of this risk classification definition.
Summary Among 48 344 working-aged Cleveland Clinic employees, those “up-to-date” on COVID-19 vaccination did not have a lower risk of COVID-19 than those not “up-to-date”. The current CDC definition provides a meaningless classification of risk of COVID-19 in the adult population.
INTRODUCTION
In April 2023 the Centers for Disease Control and Prevention (CDC) updated its guidance on Coronavirus Disease 2019 (COVID-19) vaccination to consider all individuals above the age of 6 to be “up-to-date” with COVID-19 vaccination only if they had received at least one dose of a COVID-19 bivalent vaccine [1]. By this definition, those who had not received a single dose of a COVID-19 bivalent vaccine would be considered not “up-to-date”. A recent study was unable to find the bivalent vaccine to be effective while the XBB lineages were the dominant circulating strains [2]. Given this lack of effectiveness of the bivalent vaccine against the XBB lineages, it was reasonable to question whether individuals “up-to- date” with a vaccine of questionable effectiveness were protected against COVID-19 compared to those not “up-to-date”, when the XBB lineages were the predominant circulating strains.
The purpose of this study was to evaluate whether individuals who were “up-to-date” on COVID- 19 vaccination had a lower risk of COVID-19 than those not “up-to-date”.
METHODS
Study Design
This was a retrospective cohort study conducted at the Cleveland Clinic Health System (CCHS) in the United States.
Patient Consent Statement
The study was approved by the Cleveland Clinic Institutional Review Board as exempt research (IRB no. 22-917). A waiver of informed consent and waiver of HIPAA authorization were approved to allow the research team access to the required data.
Setting
Since the arrival of the COVID-19 pandemic at Cleveland Clinic in March 2020, employee access to testing has been a priority. Voluntary vaccination for COVID-19 began on 16 December 2020, and the monovalent mRNA vaccine as a booster became available to employees on 5 October 2021. The COVID- 19 bivalent mRNA vaccine began to be offered to employees on 12 September 2022. The study start date was 29 January 2023, the date the XBB lineages first became the dominant circulating strains in Ohio.
Participants
CCHS employees in employment at any location in Ohio when the COVID-19 bivalent vaccine first became available (12 September 2022), and who were still employed when the XBB lineages became dominant (29 January 2023), were included in the study. Those for whom age and sex were not available were excluded.
Variables
Covariates collected were age, sex, job location, prior SARS-CoV-2 infection, and number of prior vaccine doses, as described in our earlier studies [2–5]. Institutional data governance rules related to employee data limited our ability to supplement our dataset with additional clinical variables. Vaccination status was “up-to-date” or “not up-to-date” based on whether or not a person had received at least one dose of a COVID-19 bivalent vaccine. For each subject, the propensity to get tested for COVID-19 was defined as the number of COVID-19 nucleic acid amplification tests (NAATs) done divided by the number of years of employment at CCHS during the pandemic. The pandemic phase during which a subject had his or her last prior episode of COVID-19 was also collected as a variable, based on which variant/lineages accounted for more than 50% of infections in Ohio at the time [6].
Outcome
The study outcome was time to COVID-19, the latter defined as a positive NAAT for Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) any time after the study start date. Outcomes were followed until 10 May 2023, allowing for evaluation of outcomes up to 100 days from the study start date. Data were accessed on 11 May 2023. Authors had access to information that could identify individual participants. However all data were de-identified before analysis.
Statistical Analysis
A Simon-Makuch hazard plot [7] was created to compare the cumulative incidence of COVID-19 in the “up-to-date” and “not up-to-date” states, by treating vaccination status as a time-dependent covariate. A subject’s vaccination status was “not up-to-date” before receipt of a COVID-19 bivalent vaccine, and “up-to-date” after receipt of the vaccine. Subjects whose employment was terminated during the study period before they had COVID-19 were censored on the date of termination. Curves for the “not up-to-date” state were based on data while the vaccination status was “not up-to-date”. Curves for the “up- to-date” state were based on data from the date the vaccination status changed to “up-to-date”.
Multivariable Cox proportional hazards regression models were fit to examine the association of various variables with time to COVID-19. Vaccination status was included as a time-dependent covariate [8]. The possibility of multicollinearity in the models was evaluated using variance inflation factors. The proportional hazards assumption was checked using log(-log(survival)) vs. time plots. The hazard ratio for vaccination status provided a comparison of risk of COVID-19 based on vaccination status after adjusting for all other covariates.
The analysis was performed by N. K. S. and A.S.N. using the survival package and R version 4.2.2 (R Foundation for Statistical Computing) [8–10].
RESULTS
Of 48 344 included subjects, 1445 (3%) were censored during the study because of termination of employment. By the end of the study, 12 841 (27%) were “up-to-date” on COVID-19 vaccination, of whom 11 187 (87%) received the Pfizer vaccine and 1654 (13%) received the Moderna vaccine. Those “up-to-date” by the end of the study included 37 of the 6174 who had previously been unvaccinated at the start of the study. A total of 1475 employees (3%) acquired COVID-19 during the 100 days of the study.
Baseline Characteristics
Table 1 shows the characteristics of subjects included in the study. Notably, this was a relatively young population, with a mean age of 43 years. Among these, 22 407 (46%) had previously had a documented episode of COVID-19 and 16 262 (34%) had previously had an Omicron variant infection. 42 160 subjects (87%) had previously received at least one dose of vaccine and 44 432 (92%) had been previously exposed to SARS-CoV-2 by infection or vaccination. Altogether, 36 490 subjects (76%) were tested for COVID-19 by a NAAT at least once while employed at Cleveland Clinic. The propensity for COVID-19 testing ranged from 0 to 63.5 per year, with a median of 0.64 and interquartile range spanning 0.32 to 1.27 per year.
Risk of COVID-19 Based on Vaccination Status and Prior Infection
The risk of COVID-19 was lower in the “not up-to-date” state than in the “up-to-date” state, with respect to COVID-19 vaccination (Figure 1). When stratified by tertiles of propensity to get tested for COVID-19, the “up-to-date” state was not associated with a lower risk of COVID-19 than the “not up-to- date” state in any tertile (Figure 2).
Simon-Makuch hazard plot comparing the cumulative incidence of COVID-19 in the “up-to- date” and “not up-to-date” states with respect to COVID-19 vaccination. Day zero was 29 January 2023, the day the XBB lineages of the Omicron variant became the dominant strains in Ohio. Point estimates and 95% confidence intervals are jittered along the x-axis to improve visibility.
Simon-Makuch hazard plot comparing the cumulative incidence of COVID-19 in the “up-to- date” and “not up-to-date” states with respect to COVID-19 vaccination, stratified by tertiles of propensity to get tested for COVID-19. Day zero was 29 January 2023, the day the XBB lineages of the Omicron variant became the dominant strains in Ohio. Point estimates and 95% confidence intervals are jittered along the x-axis to improve visibility. Solid lines represent the “up-to-date” states while dashed lines represent the “not up-to-date” states.
In contrast, consideration of prior infection provided a more accurate classification for risk of COVID-19. Those whose last prior SARS-CoV-2 infection occurred during the Omicron BQ or BA.4/BA.5 dominant phases had a substantially and significantly lower risk of COVID-19 than those whose last SARS-CoV-2 infection occurred in the early Omicron or pre-Omicron periods or those not previously known to have had COVID-19 (Figure 3). When stratified by most recent prior SARS-CoV-2 infection, there was no difference in risk of COVID-19 between the “up-to-date” and “not up-to-date” states within each most recent prior infection group, except for those not previously known to be infected, among whom the “up-to-date” state was associated with a higher risk of COVID-19 than the “not up-to- date” state (Figure 4).
Hazard plot comparing the cumulative incidence of COVID-19 stratified by the phase during which the last prior episode of COVID-19 occurred. Day zero was 29 January 2023, the day the XBB lineages of the Omicron variant became the dominant strains in Ohio. Point estimates and 95% confidence intervals are jittered along the x-axis to improve visibility. Solid lines represent those last infected before the Omicron variants became dominant or those not previously known to have had COVID-19, while dashed lines represent those last infected while the Omicron variants were dominant.
Simon-Makuch hazard plot comparing the cumulative incidence of COVID-19 in the “up-to- date” and “not up-to-date” states with respect to COVID-19 vaccination, stratified by the phase during which the last prior episode of COVID-19 occurred. Day zero was 29 January 2023, the day the XBB lineages of the Omicron variant became the dominant strains in Ohio. Point estimates and 95% confidence intervals are jittered along the x-axis to improve visibility. “Remote infection” includes infections during the pre-Omicron and Omicron BA.1/BA.2 dominant periods, i.e more than 218 days before the study start date. Solid lines represent the “up-to-date” states while dashed lines represent the “not up-to-date” states.
Multivariable Analysis
In a multivariable Cox proportional hazards regression model, adjusted for propensity to get tested for COVID-19, age, sex, phase of most recent SARS-CoV-2 infection, and number of prior vaccine doses, being “up-to-date” on COVID-19 vaccination was not associated with a lower risk of COVID-19 than not being “up-to-date” (HR, 1.05; 95% C.I., 0.88-1.25; P-value, 0.58). Having had their last prior episode of COVID-19 while lineages of the Omicron variant were dominant was protective against COVID-19 with higher protection the more recent the infection. A higher number of prior vaccine doses was associated with higher risk of COVID-19. Point estimates of hazard ratios and associated 95% confidence intervals for the variables included in the unadjusted and adjusted Cox proportional hazards regression models are shown in Table 2. If number of prior vaccine doses was not adjusted for, being “up-to-date” on COVID-19 vaccination was associated with a higher risk of COVID-19 that not being “up-to-date” (HR, 1.30; 95% C.I., 1.16-1.45; P-value, <0.001). Results were very similar if those 65 years and older were only considered “up-to-date” after receipt of at least 2 doses of a bivalent vaccine (Supplementary Table 1).
Results were also very similar if an individual was only considered “up-to-date” 7 days after receipt of a bivalent vaccine (Supplementary Table 2).
DISCUSSION
This study found that being “up-to-date” on COVID-19 vaccination, using the current CDC definition, was not associated with a lower risk of COVID-19 than not being “up-to-date”, while the XBB lineages were the dominant circulating strains of SARS-CoV-2, after adjusting for important factors that might influence risk of COVID-19.
The strengths of our study include its large sample size, and its conduct in a healthcare system that devoted resources to have an accurate accounting of who had COVID-19, when COVID-19 was diagnosed, who received a COVID-19 vaccine, and when. The study methodology, treating vaccination status as a time-dependent covariate, allowed for determination of vaccine effectiveness in real time.
Adjusting for the propensity to get tested for COVID-19 should have mitigated against bias that might arise from infections being more likely to be detected in individuals who tested more frequently.
Furthermore, those who received the COVID-19 bivalent vaccine did so with an expectation that it provided protection against COVID-19. So, if there was a difference in likelihood of getting tested before and after receiving the COVID-19 bivalent vaccine, one would expect relatively less testing in the “up-to- date” state than in the “not up-to-date” state, having the effect of undercounting infections in the “up-to- date” state, thereby falsely overestimating the beneficial effect of being “up-to-date”.
Our study was limited to examination of all detected infections. We were unable to distinguish between symptomatic and asymptomatic infections, and the rarity of severe illnesses precluded studying whether being “up-to-date” decreased severity of illness. Information on prior COVID-19 would have been incomplete, as many asymptomatic and mildly symptomatic infections would have been missed. Infections diagnosed by in-home testing without follow-up PCR testing would have been missed, but there is no reason to think these would have been missed in the “up-to-date” and “not up-to-date” states at rates different enough to change the results significantly. There may have been unconsidered variables that might have influenced the results. Lastly, our study was done in a healthcare population, and included no children and few elderly subjects, and the majority of study subjects would not have been immunocompromised.
Specific prior infections, prior vaccinations, time since proximate exposure to SARS-CoV-2 by prior infection or vaccination, number of prior vaccine doses, and interval between vaccines, all possibly contribute to determining an individual’s risk of acquiring COVID-19. It is difficult to account for the effects of all these factors in an individual. What this study clearly shows is that a simplistic categorization of being or not being “up-to-date” on COVID-19 bivalent vaccination did not provide an accurate classification of risk of acquiring COVID-19 while the XBB lineages were the dominant circulating strains. The questionable effectiveness of the vaccine, as has been previously demonstrated [2], is likely a significant contributor to the lack of accuracy of risk classification based on receipt of the vaccine.
This study’s findings question the wisdom of promoting the idea that every person needs to be “up-to-date” on COVID-19 vaccination, as currently defined, at this time. It is often stated that the primary purpose of vaccination is to prevent severe COVID-19 and death. We certainly agree with this, but it should be pointed out that there has not yet been a single study that has shown that the COVID-19 bivalent vaccine protects against severe disease or death caused by the XBB lineages of the Omicron variant. At least one prior study has failed to find a protective effect of the bivalent vaccine against the XBB lineages of SARS-CoV-2 [2]. Additionally, the finding of higher risk of COVID-19 with higher number of prior vaccine doses in this and our prior study [2], should be taken into consideration. People may still choose to get the vaccine, but an assumption that the vaccine protects against severe disease and death is not reason enough to unconditionally push a vaccine of questionable effectiveness to all adults.
In conclusion, this study found that being “up-to-date” on COVID-19 vaccination by the CDC definition was not associated with a lower risk of COVID-19 than not being “up-to-date”. This study highlights the challenges of counting on protection from a vaccine when the effectiveness of the vaccine decreases over time as new variants emerge that are antigenically very different from those used to develop the vaccine. It also demonstrates the folly of risk classification based solely on receipt of a vaccine of questionable effectiveness while ignoring protection provided by prior infection.
Notes
Author contributions
N. K. S.: Conceptualization, methodology, validation, investigation, data curation, software, formal analysis, visualization, writing- original draft preparation, writing- reviewing and editing, supervision, project administration. P. C. B.: Resources, investigation, validation, writing- reviewing and editing. A. S. N.: Methodology, formal analysis, visualization, validation, writing- reviewing and editing. S. M. G.: Project administration, resources, writing- reviewing and editing.
Potential conflicts of interest
The authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.
Funding
None.
Data Availability
All data produced are available online at https://osf.io/bufwc/
Footnotes
Number of prior vaccine doses has been added as a variable in the multivariable analysis. In the multivariable analysis, the reference level for the vaccination status has been changed to "not up-to-date". The discussion has been improved.
