Time-varying effectiveness of the mRNA-1273, BNT162b2 and Ad26.COV2.S vaccines against SARS-CoV-2 infections and COVID-19 hospitalizations and deaths: an analysis based on observational data from Puerto Rico

Background We collected hospitalization, death, and vaccination status data for all 86,488 laboratory- confirmed SARS-CoV-2 infections in Puerto Rico since the first COVID-19 vaccine was administered starting on December 15, 2020 and ending September 24, 2021. Using these data we estimated real-world time-varying effectiveness of the mRNA-1273 (Moderna), BNT162b2 (Pfizer), and Ad26.COV2.S (J & J) COVID-19 vaccines to quantify the public health benefits of Puerto Rico's immunization campaign. Furthermore, we compared the effectiveness of the COVID-19 vaccines before and after the dominance of the Delta variant. As of this writing, Puerto Rico had a higher vaccination rate and lower SARS-CoV-2 infection rate than all 50 States in the USA. Methods We used data obtained from the integration of the Puerto Rico Department of Health databases holding vaccination status, SARS-CoV-2 test results, and COVID-19 hospitalizations, and deaths. We estimated time-varying vaccine effectiveness against SARS-CoV-2 infections by fitting a statistical model that adjusts for time-varying incidence rates, age, gender, and day of the week. We also used this model to estimate the relative risk of hospitalization and deaths comparing vaccinated to unvaccinated individuals. Code and data are provided to reproduce the analysis here: https://github.com/rafalab/vax-eff-pr Results All vaccines were effective at reducing risks for all outcomes across all age groups. At the peak of their protection, mRNA-1273, BNT162b2, and Ad26.COV2.S had an effectiveness of 87% (85% - 89%), 85% (82% - 87%), and 65% (58% - 70%), with Ad26.COV2.S reaching this peak 32 days after the being considered fully vaccinated. After four months, effectiveness waned to about 73%, 58%, and 32% for mRNA-1273, BNT162b2, and Ad26.COV2.S, respectively. All vaccines had a lower effectiveness for those over 85 years, with the decrease in effectiveness particularly low for the Ad26.COV2.S vaccine. We found no clear evidence that effectiveness was different after the Delta variant became dominant. For those infected, the vaccines provided further protection against hospitalization and deaths across all age groups, although the protection was less for those above 85 years. Using the rates observed for the unvaccinated we would have observed 6,109 and 2,071 hospitalizations and deaths among the vaccinated population but we instead observed 728 and 164, respectively. Conclusions The mRNA-1273 and BNT162b2 vaccines where highly effective across all age groups. They were still effective after four months although the protection did wain. The Ad26.COV2.S vaccine was effective but to a lesser degree, especially for the older age groups, and with the protection waning substantially sooner in comparison to the other two vaccines.

Since the start of the vaccination process, the effectiveness of the COVID-19 vaccines has been estimated in several observational studies. A population-based study performed early during the vaccination process in Israel estimated effectiveness to be 72% for the BNT162b2 vaccine. 5 A case-control study, also in Israel, found 87% effectiveness for the BNT162b2 vaccine, however, lower protection against hospitalizations and deaths. 6 These estimates have been questioned due to lack of adjustment for age. A study conducted in inpatient care settings, across 187 hospitals in the United States, compared all three vaccines found the mRNA-1273, BNT162b2, and Ad26.COV2.S COVID-19 vaccines were found to be highly effective against hospitalization and death. 7 Other observational studies have estimated vaccine effectiveness in the midst of variants of concern. The effectiveness against infection in the dominance of the Delta (B.1.617.2) variant has been shown to remain at 88% for the BNT162b2 COVID-19 vaccine. 8 Similarly, in a population-based observational study found that two doses of the mRNA-1273 and BNT162b2 COVID-19 vaccines provided substantial protection against the Alpha (B.1.1.7), Beta (B.1.351), Gamma (P.1), and Delta variants of concern. 9 Nonetheless, real-world population based estimates of the mRNA-1273, BNT162b2, and Ad26.COV2.S COVID-19 vaccines effectiveness against infections, hospitalizations and deaths is still limited. Here we add to these findings by examining the effect of vaccines on all laboratory-confirmed SARS-CoV-2 infections observed in Puerto Rico since the vaccination process started, which provided enough power to estimate effectiveness across age groups and as function of time since the individuals are considerd fully vaccinated (14 days after final dose).
Following the emergency approval of the BNT162b2 vaccine manufactured by Pfizer-BioNTech on December 11, 2020, vaccine administration began in Puerto Rico on December 15, 2020. The logistics of the vaccination process proposed by the Puerto Rico Department of Health established that the vaccine administration would be completed in phases, as recommended by the World Health Organization (WHO) and the Centers for Disease Control and Prevention (CDC). 10 The phases (Supplementary Table S2) consisted on vaccinating populations that were at greater risk of infection and severe disease. For example, front line health care workers and people 65 years and older were vaccinated in the first phases.
By October 1, 2021 more than 2 million of the 3,285,874 individuals living in Puerto Rico had completed the COVID-19 vaccination series. Furthermore, since the start of the vaccination process to October 1, 2021, 111,052 cases have been detected with molecular or antigen tests in Puerto Rico with two surges, one starting in late March after restrictions were lifted and another in late June with the arrival of the Delta variant. We leveraged data collected by the Puerto Rico Department of Health to study the effect the vaccination process in preventing SARS-CoV-2 outcomes by comparing unvaccinated individulas 12 years or older to those who had completed the vaccination series (two weeks after final dose) for mRNA-1273, BNT162b2, or Ad26.COV2.S COVID-19 vaccines. In particular, we provide estimates of the effectiveness of the three COVID-19 vaccines protecting against infections, hospitalizations, and deaths to quantify the public-health impact of the vaccine in Puerto Rico.

Data sources
Two Puerto Rico Department of Health databases were integrated: the BioPortal, which stores test results, most hospitalizations, and deaths, and the Puerto Rico Electronic Immunization System (PREIS), which stores vaccination related data. For the analysis in this study we used daily counts of laboratory-confirmed SARS-CoV-2 infections, hospitalizations, and deaths from December 15, 2020 to October 1, 2021. We defined the first day to which an individual tested positive to a molecular or antigen test as the day of the infection. This same date was used for the analyses involving hospitalizations and deaths.
These daily counts were stratified by gender, age group (12-17, 18-24, 25-34, 35-44, 45-54, 55-64, 65-74, 75-84, or 85+) and vaccination status (unvaccinated, mRNA-1273, BNT162b2, or Ad26.COV2.S). Fully vaccinated was defined as 14 days after the final dose in the vaccine series. Cases in which the infection occurred after the first dose but before being fully vaccinated were removed from the analysis. Furthermore, the analysis was applied to two periods, before and after June 15, 2021, the approximate date in which the Delta variant became dominant in Puerto Rico.

Statistical analyses
We applied models that accounted for age, gender, and, since many more tests were performed on weekdays than weekends, day of the week. We used the estimates obtained from fitting these models to quantify effectiveness and relative risks. The models used for the different analyses are described below.

Time-varying effectiveness against infection by age, manufacturer
We defined Y (v) t,d,a,g as the number of laboratory-confirmed SARS-CoV-2 infections observed on day t for individuals of gender g (male or female) and age group a (12-17, 18-24, 25-34, 35-44, 45-54, 55-64, 65-74, 75-84, or 85+), that were fully vaccinated with vaccine v (mRNA-1273, BNT162b2, or Ad26.COV2.S), d days previous to the positive test. We assumed these counts followed an over-dispersed Poisson distribution with expected rate defined by Here N (v) t,d,a,g is the population size on day t of individuals of gender g, age group a that were fully vaccinated with vaccine v, d days before day t, µ a,g (t) is the incidence rate on date t among the non-vaccinated individuals of gender g and age a, and β (v) a (d) is the age-specific log relative risk of vaccine v, d days after the first dose was administered. Note the the transformation f (x) = 1 − exp(x) can be used to convert from relative risk to effectiveness. Since incidence changes across demographic groups and across days of the week (more tests performed on Mondays and less on Sundays, for example) we modeled the µ a,g (t) as with α g a gender effect, f a (t) a cubic-spline representing an age-specific trend that accounts for the change in incidence across time, and 1 w (t) = 1 indicator functions for each day of the week w ∈ {Sunday, Monday, . . . Friday}. To estimate µ a,g (t) we assumed laboratory-confirmed SARS-CoV-2 infections counts for non-vaccinated individuals, denoted with Y (u) a,g , follow a Poisson distribution with rate µ a,g (t) and obtained the MLE for all parameters using Iteratively Reweighted Least Square algorithm, as implemented by the glm function in R. We then used these to define the MLE for µ a,g (t) and denoted it witĥ µ a,g (t). With this estimate in place we then estimated the parameter of interest β (v) a by treating µ a,g (t) in (1) as a known value, plugged inμ a,g (t), and noting that t Y (v) t,d,a,g follows a Poisson distribution with expected value 3 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted October 18, 2021. ; Because the first term on the right is treated as known, if we assume β (v) a (d) is a cubic-spline, then the model defined by (2) is reduced to a simple generalized linear model and we find the MLE β (v) a (d) again using Iteratively Reweighted Least Square.

Relative risk of hospitalization and death
Note that we can use this model for the hospitalization and deaths outcomes by simply redefining the counts Y (v) t,d,a,g based on counts of these outcomes. Since we observed less hospitalization and death data than for infections, to increase power we also ran an analysis that assumed β (v) a (d) was constant across d and the same across age groups. Specifically we assumed that the age groups 18-44, 45-74, 75-84, and 85+ shared the same β (v) a (d). We then fit model (1) to obtain estimates for the relative risks.

Hospitalization and death relative risk conditioned on infection
Note: in this section, to avoid introducing more mathematical symbols, we repurpose the Y, N, α, β and γ notation and, because we fit a model separately to each age group, we do not use the a index.
To estimate the further protection provided by the vaccine in reducing hospitalization and deaths among infected individuals we examined the proportion of cases that had hospitalizations or deaths. To do this we defined N i,t as the number of laboratory-confirmed SARS-CoV-2 infections on day t for group i. Groups were defined by gender (male or female) and vaccination status (unvaccinated, mRNA-1273, or BNT162b2). Note that we did not include the Ad26.COV2.S vaccine in this analysis due to small sizes obtained after stratifying by age group. We defined Y i,t as the number of these that had the hospitalization or death outcome. We then assumed that Y i,t followed a binomial distribution with N i,t trials and success probability p i,t defined by: with β 0 the baseline log odds for the unvaccinated, X i,1 = 1 if group i was vaccinated with Ad26.COV2.S and 0 otherwise, X i,2 = 1 if group i were vaccinated with mRNA-1273 and 0 otherwise, X i,3 = 1 if group i are male and 0 otherwise, and 1 w (t) indicator functions as in (1). We fit this model using the Iteratively Reweighted Least Square algorithm. Note that the parameters of interest were β 1 and β 2 .

Vaccination data
As of October 1, 2021, 2,217,547 of the 3,285,874 individuals living in Puerto Rico had been fully vaccinated: 1,243,969 with BNT162b2, 844,065 with mRNA-1273, and 129,513 with Ad26.COV2.S. The daily COVID-19 vaccination rates were high during the first four months of the vaccination process, successfully vaccinating (at least one dose) 50% of the population eligible for inoculation at the time (1,442,459 of 2,848,293). 11 However, as observed in previous vaccination campaigns, vaccine rates decreased after reaching 50%. The plateau was particularly noticeable among the older age groups. After several vaccine mandates were implemented in August and September, the rate commenced increasing ( Figure 1). With the exception of pediatric population 12 years and older, whom were only vaccinated with BNT162b2, the age distribution of the vaccine was similar for all vaccine manufacturers (Supplementary Figure S1).

4
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted October 18, 2021. ;

Observed cases, hospitalizations and deaths
During the period considered in this analysis (December 15, 2020 to October 1, 2021), 87,399 laboratoryconfirmed SARS-CoV-2 infections were observed for individuals 12 years or older. These were mostly distributed across two surges ( Figure 2). The first of these started shortly after several restrictions were lifted on March 15, 2021. Restrictions were imposed again on April 9, 2021 and the cases decreased to a level not seen since the summer of 2020. As a result, restrictions were lifted once again on May 24, 2021 and shortly after the arrival of the Delta, a second surge began by the end of June 2021. 12 The SARS-CoV-2 infections observed during the period of December 15, 2020 to October 01, 2021 resulted in at least 4,942 and 1,333 COVID-19 related hospitalizations and deaths, respectively ( Figure 2 and Supplementary Table S3).
The infections, hospitalization, and death rates were noticeably lower among the vaccinated (Supplementary Table S3). The protection afforded by the vaccine was observed by simply plotting the infection rates by vaccination group (Figure 2). The relative risk for the vaccinated seemed to increase in the period in which the Delta variant dominated (Supplementary Figure 2). However, these rates are not age adjusted nor are potential effects of waning effectiveness considered. Therefore, to better assess COVID-19 vaccine effectiveness, we fit statistical models that account for time since the first dose, age, gender, and other potential confounding variables.

Time-varying vaccine effectiveness against infections
We fit a statistical model (described in detail in the Methods Section) to estimate time-varying effectiveness while accounting for the difference in demographics between vaccinated and unvaccinated groups. Note that vaccine administration for individuals between 12 and 17 years started on May 12, 2021 and only with the BNT162b2 vaccine.
To evaluate whether the Delta variant affected vaccine effectiveness, we considered two periods: before and after June 15, 2021. The mRNA-1273 and BNT162b2 vaccines remained steady at over 75% effective after the Delta variant gained dominance and a decrease in vaccine effectiveness was not detected due to the Delta variant (Supplementary Figure S2). Furthermore, to evaluate if there were differences in effectiveness across age groups for the mRNA-1273 and BNT162b2 vaccines we ran a similar analysis, but comparing age groups. We did not observe substantial differences in effectiveness against infections by age, except for those 85 and older for whom the waning appears worse (Supplementary Figure S3). Note that this group was smaller and resulted in less precise estimates.
Considering that no clear differences were observed across age groups and time-periods, we estimated one time-varying effectiveness across all age-groups and for the entire vaccination process period. Note that the period of evaluation was different for the different age groups given the previously described phases (Table  S2). Furthermore, for the Ad26.COV2.S vaccine, the period after the first dose administration date was shorter because this vaccine received its EUA on March 3, 2021.
We fit the same model to estimate relative risk of hospitalization comparing vaccinated and unvaccinated individuals across time since fully vaccinated. Although we did not have enough data to obtain as precise estimates as for infections, similar patterns were observed (Supplementary Figure S4).

Vaccines lower the risk of hospitalization and death
We fit a statistical model (described in detail in the Methods Section) to determine if the vaccines provided further protection against hospitalization and death among infected individuals. Specifically, we estimated 5 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted October 18, 2021. ; the probability of hospitalization and death conditioned on individuals testing positive for SARS-CoV-2 for several age groups (Figure 4 and Supplementary Table S1). Due to the small sample sizes available for the Ad26.COV2.S vaccine, after stratification by age, we did not include it in this analysis. The probability for hospitalization after infection for those that were vaccinated was at least twice lower for all age groups except the 85 and older for which the probability was over 50% lower. Similarly, the probability of death after infection was over three times lower for all age groups, except the 85 and older for which it was about twice as low.
We also fit a statistical model to estimate the relative risk of COVID-19 hospitalization and death comparing vaccinated and non-vaccinated individuals (not conditioned on being infected). Due to sparsity in the data we grouped age groups as described in the Methods Section. Individuals vaccinated with mRNA-1273 or BNT162b2 between 45 and 74 years were found to be 18 (15, 22) and 8.0 (7, 9.2) times less likely to be hospitalized due to COVID-19 in comparison to unvaccinated individuals, respectively. Similarly, Ad26.COV2.S vaccinated individuals between 45 and 74 years had 3.9 (3, 5.0) times less risk for hospitalizations in comparison to the unvaccinated. For those 75 and older vaccinated with mRNA-1273 or BNT162b2, their risk for hospitalization by COVID-19 is reduced by 2.8 (2, 4.0) and 2.9 (2, 4.2) times, respectively, in comparison to unvaccinated individuals. We did not find statistically significant evidence that the Ad26.COV2.S vaccine provided added protection against hospitalization for people 85 and older ( Table 1).
All three vaccines provided high protection against death by COVID-19 for those between 45 and 84 years. For people 85 years vaccinated with mRNA-1273 or BNT162b2, the risk reduction was 3.8 (2, 6.1) and 4.8 (3, 8.5), respectively, showing these vaccine provide great protection against death for this age group. The Ad26.COV2.S vaccine did not provide added protection against death by COVID-19 for people 85 and older (Table 1).

Discussion
Our analyses clearly demonstrates that all vaccines were effective at reducing risks of infection, hospitalization and death across all age groups, with the protection provided by the and mRNA-1273 and BNT162b2 vaccine particularly strong. At the peak of their effectiveness mRNA-1273 and BNT162b2 were 90% (88% -91%) and 87% (85% -89%) effective, respectively. Meanwhile Ad26.COV2.S was 58% (51% -65%) at its peak. The effectiveness of all vaccines decreased over tine, although mRNA-1273's effectiveness remained high at 71% even after four months. BNT162b2 was also relatively high, at around 56%, however, Ad26.COV2.S effectiveness dropped to 27%. All vaccines had a lower effectiveness for those over 85 years, with the decrease in effectiveness particularly low for the Ad26.COV2.S vaccine. No evidence was found that effectiveness was different after the Delta variant became dominant.
It is important to note that this study is observational based on real world data and not a randomized clinical trial. Although we controlled for confounding factors that were coded and available for analyses, it is possible that there were unknown and unobserved confounders that were not accounted for, such as chronic health conditions and immunocompromising conditions. However, the findings described here agree with general trends observed in the clinical trials. 2 3 . 4 Following CDC guidelines, in Puerto Rico it is general practice to recommend diagnostic testing to those who are experiencing symptoms and/or are direct contacts of a case. 13 This implies there is a risk for selection bias given the fact that vaccinated people are less likely to experience severe illness or symptoms, and thus less likely to get a diagnostic test. 14 It is also important to note that to perform this analysis two Puerto Rico Department of Health databases were integrate: the BioPortal, which stores test results, hospitalizations, and deaths, and PREIS, which stores vaccination data. These two databases lacked a common unique identifier and records were matched using pacient information. This implies that some parings may have been missed, although a human spot check found the paring to be 99% accurate. We also note that due to deficiencies in reporting within the Department of Health, not all hospitalizations and deaths were included in the BioPortal database and there is lag in reporting in some vaccination centers . However, no clear reason that could bias results to lower 6 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted October 18, 2021. ; or higher vaccination effectiveness estimates was detected. Finally, it is also important to note that all our analyses depend on age specific population estimates provided by the United States Census Bureau and these are likely to change with the finalization of the 2020 census.  9 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 10 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted October 18, 2021. Probability of death among infected

11
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted October 18, 2021. ; https://doi.org/10.1101/2021.10.17.21265101 doi: medRxiv preprint Tables   Table 1: Reduction in risk provided by the different vaccines for individuals across age groups. We estimated the relative risk of hospitalization and death for these two age groups and each vaccine manufacturer, with a model that adjusted for population size, age, gender, and time varying incidence rates. Columns 3, 4, and 5 show estimated reduction in risk and 95% confidence intervals, comparing unvaccinated to vaccinated rates. Columns 6, 7, and 8 show estimated expected outcomes based on estimated unvaccinated rates and the difference between these.  Figure S1: Total number of fully vaccinated individuals by age group and vaccine manufacturer. Each pane represents a vaccine manufacturer and colors represent the different age groups. 13 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)  Figure S3: Waning effectiveness against infection by age group and vaccine manufacturer (colors). The ribbons represent point-wise 99% confidence intervals.
14 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.   Secretary of Health signs executive order authorizing the vaccination of personel in food industry, drug companies, medical equipment, the public transport sector, air transport and maritime cargo. March 29, 2021

Supplementary Tables
Phase 1C begins with the vaccination of people 50 years and older and 35 and older with cronic conditions. April 12, 2021 Phase 2 begins with vaccination available to everyone 16 years and older.
April 16, 2021 Vaccine rollout is expanded to include everyone 16 years and older. May 12, 2021 CDC reccomends vaccination for people 12 years and older.  16 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted October 18, 2021. ; https://doi.org/10.1101/2021.10.17.21265101 doi: medRxiv preprint