When time-dependence in disease outcome risk is not captured by impact evaluation modeling studies: a measles vaccination case study

Background: In modeling studies that evaluate the effects of health programs, the risk of secondary outcomes attributable to infection can vary with underlying disease incidence. Consequently, the impact of interventions on secondary outcomes would not be proportional to incidence reduction. Here we use a case study on measles vaccine program to demonstrate how failure to capture this non-linear relationship can lead to over- or under-estimation. Methods: We used a published model of measles CFR that depends on incidence and vaccine coverage to illustrate the effects of: (1) assuming higher CFR in 'no-vaccination' scenarios; (2) time-varying CFRs over the past; and (3) time-varying CFRs in future projections on measles impact estimation. We evaluated how different assumptions on vaccine coverage, measles incidence, and CFR levels in 'no-vaccination' scenarios affect estimation of future deaths averted by measles vaccination. Results: Compared to constant CFRs, aligning both 'vaccination' and 'no-vaccination' scenarios with time variant measles CFR estimates led to larger differences in mortality in historical years and lower in future years. Conclusions: To assess consequences of interventions, impact estimates should consider the effect of 'no-intervention' scenario assumptions on model parameters to project estimated impact for alternative scenarios according to intervention strategies and investment decisions.


Introduction
Model-based estimation has been widely used to evaluate the impact of infectious disease 3 intervention programs outside of empirical observations [1]. There are myriad policy interests in 4 both retrospective program evaluation to estimate the effect of a previously implemented program 5 and prospective program evaluation to project the impact of different intervention options in the 6 future. In both types of program evaluation, the program impact is often quantified by comparing 7 the estimated effects in the scenario with the intervention against those in a scenario without the 8 intervention (e.g., with and without vaccination). Under this framework, detailed methodological 9 considerations defining both the "intervention" and "no-intervention" scenario are a necessary 10 condition to minimize bias in program effect estimates. 11 12 The most common approach in developing the "no-intervention" scenario is to "switch off" the 13 program in the model. In regression models, this "switch off" can be incorporated by including an 14 indicator variable for program implementation. In mechanistic simulation models, it can be 15 modeled by setting the uptake of the intervention to zero in scenarios without program 16 implementation while keeping all other parameters consistent with scenario(s) where the program 17 is implemented. For both types of models, we can evaluate the intervention impact as the difference 18 in the outcome of interest, such as the number of deaths under the intervention and no-intervention 19 scenario. Many health impact models assume risks of infectious disease health outcomes 20 conditional on infection, such as case-fatality ratios (CFRs), are independent of disease incidence 21 and health system characteristics. However, there is evidence that the outcome of infection and 22 consequences of infection can depend on health system burden [2][3][4][5][6][7][8]. 23 24 4 An example of such an intervention is measles vaccination. Due to limited primary data of measles 25 CFR, for many years, the impact of measles vaccine has been estimated by simulation models that 26 assume constant measles CFR over time [9,10]. However, a recently updated meta-analysis [5] 27 showed a decreasing trend in measles CFR in low-and middle-income countries (LMICs) and 28 found that this is associated with trends in measles vaccination coverage [11], measles incidence 29 [5], and under-five mortality [12]. In most settings, these factors have varied over time and may 30 continue to change in the future, either continuing their past trends or potentially reversing 31 direction following the COVID-19 pandemic [13,14].

33
Consequently, in modeling studies that aim to evaluate the impact of a measles vaccination 34 program, changes in the health system is an important aspect to consider as it is related to the risk 35 of disease outcome. As disease management and health system capacity improves over time, we 36 would expect risks of disease outcome to improve over time. Likewise, changes to the population-37 level risks of disease outcome could be negatively impacted by changes to preventive measures 38 such as vaccination coverage. These observations motivate the design of this study to evaluate the 39 consequences of more realistic assumptions that affect model predictions in the "vaccination" and 40 "no-vaccination" scenarios, accounting for time-dependent elements.

42
Using measles as a case study, we propose a methodological innovation to model CFR 43 dynamically, which addresses both: (1) dependence of CFR on incidence and other health system 44 characteristics; (2) calculating impact with due consideration for the "no-vaccination" scenario. 45 This paper is organized into two parts. In Part I, we used a log-linear regression model to evaluate 46 three different sets of scenarios where covariates used to estimate measles CFR can be time variant 47 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint or time invariant. As a baseline, we used the current practice of assuming time and strategy 48 invariant CFRs as described in Wolfson, et al. [10]. We also considered two other scenarios in 49 which CFRs depend on covariates that change over time, as informed by our previous study where 50 we showed that CFRs have changed over time from 1980-2006 [5]. First, we only allowed for  We used a previously published log-linear projection model relating CFR to measles incidence, 64 time, and other factors, [5]  = an approximation of measles attack rate (estimated measles incidence divided by 74 annual birth cohort) [12,15]; 5 = all-cause under-five mortality rate per 1000 live births [12]; 75 = population density per square kilometer of land area [12]; = total 76 fertility rate [12]; and = percentage of population living in urban areas [12]. The log-linear model was subsequently used to estimate future CFRs from 2019-2030 in 94 "vaccination" and "no-vaccination" scenarios in the second part of our analyses. We used 95 projected data for covariates, including under-five mortality rate, total fertility rate, percentage of 96 population in urban areas, and population density [18]. Population density was available with 97 annual projections, whereas under-five mortality, total fertility rate, and urban percentage were 98 available by five-year increments [18]. Future MCV1 coverage was projected by the authors from 99 2018 WHO-UNICEF estimates of coverage [11], assuming a 1% coverage increase per year in 100 line with prior analyses [19]. Future MCV1 coverage was capped at 95%, unless a country had a  We estimated measles CFRs in three scenarios (Table 1). In Scenario-0, we relied on previous 107 estimates of measles CFRs [9], based on a descriptive analysis [10], assumed to be constant across correlation between CFR, vaccination coverage, measles incidence, and under-five mortality as 112 outlined in previous analysis [5]. In this scenario, the 2018 CFR estimate was assumed to be  CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint In Scenario-2, we extended the CFR projection end year from 2018 to 2030, using projected data 117 for covariates, described above. The incidence estimates from 2019 to 2030 were generated from 118 the same measles transmission models used in Scenario-1.

120
Subsequently, we used the CFR estimates to quantify the impact of estimated CFRs by analytic 121 scenario on estimates of measles deaths from 2000 to 2030, described in Part II, using the PSU 122 and DynaMICE models of measles transmission [9,20]. We used R statistical software, version 123 3.6.1, for all analyses [21]. absence of vaccination, we tested two alternative approaches. Specifically, we compared the 130 analytic scenarios 1 and 2 in Table 1 to two alternative "no-vaccination" scenarios that assumed: 131 (a) CFRs remain constant; and (b) time-varying CFRs according to the approach used in the 132 comparator "vaccination" scenario. We evaluated how the different assumptions in the "no-133 vaccination" scenarios affected our estimation on future deaths averted by measles vaccination.

134
These impact estimates were compared to impact estimates where age-specific CFRs in the "no-135 vaccination scenario" were assumed to be the same as in the corresponding "vaccination" scenario.

139
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

140
The Scenario-0 CFRs were stratified by age (</>=5 years) for each country; on average across all 141 LMICs, these CFRs were 2.1% for children less than five years of age and 1.0% for children five 142 years of age and older. In comparison, the estimated time-varying CFRs ranged from 3.7% (2.3-   Impact of matching CFR estimation approach in "no-vaccination" scenario 162 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint 163 Scenario-0 assumed constant CFRs under both the "vaccination" scenario and the "no-164 vaccination" scenario. Scenario-1 and Scenario-2 assumed estimated CFRs according to the time- 165 varying, incidence-based methodology for both the "vaccination" scenario and the "no- were more than Scenario-0 (92.0%) which is more than Scenario-2 (91.9%) in the DynaMICE 170 model (Table 3). Figure 2 displays these results graphically. In this study, we illustrated the effect of important considerations for estimating measles CFRs in 176 the evaluation of measles vaccination impact through alternative "vaccination" and "no-177 vaccination" scenarios. Our aim was to provide evidence that when measles CFR is dependent on 178 factors such as the incidence of measles and the presence of vaccination, the impact of the vaccine 179 program on mortality risk would depend on these context as well. In order to reflect this context 180 dependence, estimates of measles CFR should be dynamic in time and reactive to differences in 181 scenarios with a transparent methodology that can produce reproducible estimates. Assuming 182 constant measles CFRs produces impact (number of measles cases averted) that grows over time 183 (due to the growth in cases as population grows), but may overestimate the number of deaths that 184 can be averted by measles vaccination in prospective program evaluation and underestimate deaths 185 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. averted by measles vaccination historically and devalue past gains in retrospective program 186 evaluation. On the other hand, assuming CFRs that decline in the future, in a way that is consistent 187 with empirical observation, leads to impact that is decreasing in the future (because CFRs decline 188 faster than population growth). Recognizing this trend may provide an opportunity to capitalize on 189 these improvements in order to accelerate that decline and create a world in which no (or very few) 190 children die of measles.  In addition, these reductions reflect measles vaccination impact across 112 LMICs, representing a 208 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint difference of 10-18 million deaths averted. When matching the CFR estimation approach between 209 the "no-vaccination" and "vaccination" scenarios in Figure 2, the rebound effect in estimated 210 measles deaths seen in Scenario-1 holding CFRs constant beyond 2018 is due to increasing 211 population growth, despite the constant CFRs.  [26]. However, in methods that make one-step ahead forecasts such as ARIMA models, 230 long-term forecasting may not be reliable.

231
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.  There are several limitations to this analysis. First, the functional form for the relationship between 244 the analyzed covariates and measles CFRs is informed by a limited data set of varying quality, as 245 described previously [5]. The Immunization and Vaccine-related Implementation Research 246 Advisory Committee (IVIR-AC) of the WHO in its recent recommendations noted the need for 247 ongoing primary data collection, including "investments in strengthening outbreak investigation 248 and evaluation activities to generate additional primary data" and the "creation of a standard CFR 249 study protocol and a structured data collection tool to improve comparability of studies" [27]. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint on measles CFRs [5]. We addressed these limitations by estimating two different versions of the 255 "no-vaccination" scenario. While the causal effect of variables not included in the model might be 256 captured in the variables we included, it is unclear which variables should apply to the "no-257 vaccination" case. Third, there may be additional uncertainty in the impact estimates according to 258 the assumptions of each measles transmission model, described previously [9,20,28,29].  is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint Table 2 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint Table 3 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021 . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. As measles is a highly transmissible childhood infection, disease dynamics are inextricably linked to population structure and demographic parameters. To enable precision in the estimation of disease burden and the contact processes that drive transmission, the model is age-stratified to include weekly age groups from birth to 3 years of age, and yearly age-groups from 3-100 years . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint either be susceptible (S) to measles, infected (I) or recovered (R) with life-long immunity. After a certain duration of maternal immunity, births replenish the pool of susceptibles that in the absence of vaccination fuel periodic outbreaks of measles driven by the magnitude of the birth rate and the strength of seasonality in transmission parameters. Susceptibles get infected through contact with infected individuals, with mixing determined by age-dependent contact patterns. The contact matrix used for this exercise is the POLYMOD contact matrix for Great Britain 5 as it was best able to reproduce transmission dynamics across a range of countries, but this can be updated to represent local population age structure. Following infection, individuals either recover and gain lifelong immunity, or die as described by country-specific age-dependent case fatality ratios (CFRs).
Routine vaccination is modelled through first-and second-dose measles-containing vaccine (MCV1 and MCV2) schedules (corrected for the right cohorts) with the additional option of including SIAs. Vaccines are assumed to be "all or nothing" with effectiveness equal to 84% for the first dose among children under the age of one year, 93% for the first dose among children over the age of one year, and 99% for both doses, 6 with life-long vaccine-induced immunity. 5  . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

Appendix C. PSU Model Overview 7
The PSU measles model is a semi-mechanistic, age-structured, discrete time-step, annual SIR model. Unlike conventional SIR models, which describe dynamics at the scale of an infectious generation (TSIR REF) 8 or finer (basic REF) 9 , it models the aggregate number of cases over one- . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint The parameters 0, , 1, , and 2 are fit to each country independently using a state-space model fitted to observed annual cases reported through the JRF from 1980-2016 as described by Eilertson, et al. 10 Historical population and vaccination coverage values are provided by WHO as described by Simons,et al. 11 The number of susceptible individuals in each single-year age class a (a=2,…, 25) is equal to the number not infected in the previous year, nor immunized through supplemental immunization activities (SIAs). The number susceptible is further deprecated by the crude death rate. The efficacy of doses administered through SIAs is assumed to be 99%. The number of susceptible individuals in age class a=1 is assumed to be 50% of the annual live birth cohort; this assumes that all children have protective maternal immunity until 6 months of age. Age class a=2 and a=m is assumed to receive a first and second dose (respectively) of routine measles vaccination before the start of the time step; thus, the number susceptible is further reduced by the product of the coverage and the efficacy. Efficacy is assumed to be 85% and 93% for the first dose in countries delivering at 9m and 12m of age, respectively, and assumed to be 99% for the second dose.
Deaths are calculated by applying an age-and country-specific case fatality ratio (CFR) to each country. CFRs for cases below 59 months of age for all countries were taken from Wolfson, et al. 12 ; CFR for cases above 59 months of age are assumed to be 50% lower than those applying to under-5s. 10 Eilertson KE, Fricks J, Ferrari MJ. Estimation and prediction for a mechanistic model of measles transmission using particle filtering and maximum likelihood estimation. Statistics in Medicine 2019; 38:4146-58. 11  . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint Forward simulations of this model assume random variation in the annual attack rate according to the parameter 2 . Further, each forward simulation draws 0, , 1, at random from the joint 95% interval estimate of each parameter. Future vaccination coverage values, for routine and SIAs, are assumed known and future birth and death rates are assumed known.
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 15, 2021. ; https://doi.org/10.1101/2021.07.12.21260376 doi: medRxiv preprint