Risk Heterogeneity and the Illusion of Waning Vaccine Efficacy

Michael P. Fay et al.

Critical scientific questions about COVID-19 vaccines are whether the vaccine effect will wane over time, whether they will work on new variants, and whether booster shots are needed. As reports of the durability of vaccine protection become available (1–4), general scientific readers should be aware of a subtle issue, known in the statistics literature as frailty effects or in the vaccine literature as depletion-of-susceptibles bias (5–8). The issue is that a study with higher- and lower-risk participants (in other words, risk heterogeneity) can create an illusion of waning vaccine efficacy (VE) over time even when the effect of the vaccine on each individual is not changing, or it can exaggerate the estimated magnitude of true individual waning VE.

Vaccine efficacy is defined as the percentage reduction in disease incidence among a vaccinated group compared with an unvaccinated group over a specified time period. It is typically measured in 2 previously uninfected populations: one newly vaccinated and the other not (the control population). Randomization at baseline is used to balance the risk factors for future infection. As an example, if 10% of a control group and 7% of a vaccinated group get the disease over 10 months of follow-up, the VE over the 10-month period is 30% (1 − 7%/10%). A complexity arises when we want to evaluate how VE changes over time, such as comparing efficacy for the first 5 months compared with the second 5 months.

When measuring VE at different time periods since vaccination, we typically exclude participants who have gotten the disease in earlier time periods in our calculations, because we want to estimate the vaccine’s effect on people who have not yet experienced the disease within the study. A complicating issue is that people differ in immune functionality (some individuals are more frail and have a less robust immune system) and exposure risk (for example, some individuals may have high risk, such as health care workers or nursing home residents). Randomization creates 2 populations (vaccine and control) with the same proportions of frail and/or high-risk individuals at the start of the studies but, if the vaccine works, in the later periods there will be lower proportions of frail/high-risk individuals left in the control group who did not have the disease in early periods.

Consider a simple example in the Figure, based on a model with 2 types of individuals: low-risk individuals and high-risk individuals—where the risk for disease is 4 times greater for the high-risk group. In the example, an individual’s VE is about 75% regardless of risk group or period. At randomization, the control and vaccine groups have equal proportions of high-risk participants, so the early population VE estimate is 75%, and it is just the average of individual VEs. But in the late period, many fewer high-risk participants are in the control group, because they did not make it disease-free to the late period. Unlike the start of the study, we are now comparing “apples,” or higher-risk vaccine recipients, to “oranges,” or lower-risk control participants. Thus, a smaller proportion of control participants get the disease in the later period. The late period population VE estimate is 62%. The individual VE has not changed over time, and the only difference between the early and late periods is the increased proportion of low-risk people in the control group. This can happen in big studies, too; if each figure in the graph represented 1000 people, the differences in population VE from early to late periods remain. The problem should not markedly affect a study with a low percentage of events (for example, Polack and colleagues [9] had an event rate of <0.5% for the initial VE estimates) because few people get the disease in the early period of the study. For large real-world observational studies, the problem may be more severe because the risk groups are not balanced between unvaccinated and vaccinated participants.