Reply to Fung et al. on COVID‐19 vaccine case‐counting window biases overstating vaccine effectiveness
Raphael Lataster
Abstract
Kaiser Fung, Mark Jones, and Peter Doshi1 raised some important points in their recent article published here in the Journal of Evaluation in Clinical Practice, on potential biases that could be exaggerating the effectiveness of the messenger RNA (mRNA) COVID-19 vaccines. In this reply paper, I highlight how an example they provided can be improved, in order to avoid underestimating the effect of their proposed case-counting window bias, and expand on their discussion around the potential impacts of case-counting window biases. Fung et al.1 correctly explain that in the clinical trials, “Investigators did not begin counting cases until participants were at least 14 days (7 days for Pfizer) past completion of the dosing regimen, a timepoint public health officials subsequently termed ‘fully vaccinated’”. They also note that “The rationale for excluding cases occurring before the start of this ‘case-counting window’ was not provided”. The problem is that outside of clinical trials, the effective placebo group does not exclude cases, leading to an “asymmetry, in which the case-counting window nullifies cases in the vaccinated group but not in the unvaccinated group”, leading to biased and inaccurate estimates. They present an example, in their Table 1, detailing how this bias can give the perception that a completely ineffective vaccine is 48% effective. As expected, the result is now more pronounced, with a vaccine with 0% effectiveness perceived as having 65% effectiveness. While this is a conspicuous result, it might actually understate the effect of a case-counting bias, at least in relation to COVID-19. Fung et al.1 made no errors in the first part of their calculations, concerning clinical trials, revealing the case rates to be identical in the vaccinated and unvaccinated groups, as is to be expected for a vaccine with 0% efficacy. However, for the second part of their calculations, concerning observational studies, they used symptomatic cases during the case-counting window in calculating the case rate for the vaccinated, but all symptomatic cases in the unvaccinated in calculating the case rate for the unvaccinated. At first glance this seems reasonable, however, their apparent assumption of “nullified cases” in the vaccinated group overlooks the fact that the symptomatic cases in the vaccinated that occurred before the case-counting window do not necessarily disappear from the data altogether but could rather be added to the figures for the unvaccinated, a form of definitional bias. The extent of this would vary by study, with one example coming from the UK government's (via the ONS) definition of unvaccinated in one report as being “those with no vaccination or who were vaccinated with a first dose less than 21 days ago”.2 Similarly, in the United States the CDC produced a report stating that the “incidence of SARS-CoV-2 infection, hospitalization, and death is higher in unvaccinated than vaccinated persons”, with a footnote explaining that, “Fully vaccinated persons are those who are ≥14 days postcompletion of the primary series of an FDA-authorised COVID-19 vaccine. Not fully vaccinated persons are those who did not receive an FDA-authorised COVID-19 vaccine or who received vaccine but are not yet considered fully vaccinated.”3 To show the potential magnitude of such a definitional bias, I reproduce an altered version of Table 1 from Fung et al.,1 revealing the extent of the bias if all vaccinated cases occurring before the case-counting window were not simply ignored but added to the unvaccinated group in observational studies (Table 1). One counterintuitive development since the mRNA vaccines have been rolled out is the occasional observation of “negative effectiveness” in several studies, varyingly for infection, hospitalisation, and death. As but one example, Goldberg et al.4 examined COVID infection in various groups, finding that the “adjusted rate of confirmed infections among recovered, unvaccinated persons 4 to less than 6 months after infection was 10.5 per 100,000 person-days at risk (95% confidence interval [CI], 8.8 to 12.4); this rate increased to 30.2 (95% CI, 28.5 to 32.0) among persons in this cohort 12 months or more after infection”. By comparison, for “the two-dose cohort, the rate was 21.1 (95% CI, 20.0 to 22.4) among persons vaccinated within the previous 2 months, and this rate increased to 88.9 (95% CI, 88.2 to 89.5) among those vaccinated 6 to less than 8 months previously”.4 There are a multitude of potential explanations, such as that the unvaccinated benefitted from a superior natural immunity, or that the vaccine temporarily renders vaccinees immunocompromised, or due to original antigenic sin the initially effective vaccine actually is negatively effective for later variants. On the latter, and in light of the Fung et al.1 paper, and my addition, it appears that this case-counting window bias, along with the associated definitional bias, could help explain why the effectiveness of the vaccines appears to wane so quickly and even apparently becomes negative. To illustrate, I reproduce another altered version of the example in Fung et al.,1 to demonstrate how a vaccine with effectiveness of −100% for a particular variant can be perceived as being effective (Table 2). Just as in the earlier example, we observe a substantial difference between the actual effectiveness and the perceived effectiveness, due to the case-counting window and definitional biases. In the Fung et al.1 example, a vaccine with 0% effectiveness is perceived as having 48% effectiveness; or 65% with my revision. And here, a vaccine with −100% effectiveness, meaning that it makes symptomatic COVID-19 infection twice as likely, can be perceived as being 47% effective. And this is before we also account for the age and background infection rate biases also discussed in Fung et al.,1 though that is beyond the scope of this reply. This effect would be larger still, if a greater proportion of the apparently vaccine-caused cases occurred before the case-counting window. The effect of considering cases in the vaccinated as being in the unvaccinated is so marked that even a vaccine with −200% effectiveness, meaning that it makes symptomatic COVID-19 infection three times as likely, can be perceived as being 36% effective (Table 3). Repeated calculations will show that moderate vaccine effectiveness is still perceived even with actual vaccine effectiveness figures of −1000% and lower. Please note that further calculations, without accounting for definitional biases, as in Fung et al.,1 still demonstrate that vaccines with moderate negative effectiveness can be perceived as being moderately effective. Fung et al.1 limited their article to effectiveness. However, I wish to quickly elaborate on how the case-counting window bias, and the definitional bias, could apply to safety monitoring as well. While it may in some circumstances be appropriate to monitor the effectiveness of the mRNA vaccines from the point that they are most effective, whether it be 7, 14, or 21 days from the completion of the vaccination programme (so long as the cases occurring before the window are accounted for somehow), there is no sound rationale for this to apply to safety analyses. Ignoring adverse effects of the vaccine that occurred before the case-counting window could drastically underestimate the risks of the vaccines, particularly as public health officials agree, such as at the CDC, that serious health problems tend to occur early, typically “within six weeks of getting a vaccine”.5 Definitional biases would also apply if adverse effects caused by the vaccine but occurring before the case-counting window would be labelled as unrelated incidents in what is described as an “unvaccinated person”. This is not necessarily occurring now but these biases could become an issue in future studies, as public concerns grow over the mysterious rise in non-COVID excess deaths post-pandemic,6, 7 and some researchers may wish to compare the overall health outcomes of the vaccinated and unvaccinated, to rule out the mRNA COVID-19 vaccines as playing a significant role. This is already happening in the United Kingdom to some extent, with another ONS report focussing on overall deaths in England by vaccination status. They did well in utilising an “ever vaccinated” category (meaning that “unvaccinated” people had exactly 0 doses), though a note states, “There were some people who were vaccinated but not included in the NIMS data as they died soon after vaccination.”8 Any report relying primarily on the NIMS data then may as a result be excluding vaccine adverse effects, and these may also end up being considered as unrelated events in the unvaccinated population. It is thus crucial to keep these biases in mind when carrying out observational studies going forward. None of this is to say that the mRNA COVID-19 vaccines are ineffective or unsafe. Rather, in the interests of obtaining the most accurate figures when measuring efficacy and safety, it is imperative that researchers are aware of the potential effects of case-counting window and definitional biases, and consider making a distinction between how participants are grouped with regards to measuring effectiveness and how participants are grouped with regards to safety monitoring. Fung et al.1 aided greatly to the discussion about mRNA COVID-19 vaccine effectiveness, by highlighting the potential effect of biases such as case-counting window biases. I further stress the importance of these biases, noting their implications on safety monitoring as well as perceptions of effectiveness, whilst also explaining that the effects of these case-counting window biases are substantially increased when accompanied by definitional biases. Open access publishing facilitated by The University of Sydney, as part of the Wiley - The University of Sydney agreement via the Council of Australian University Librarians. The author declares no conflict of interest. Data sharing not applicable to this article as no datasets were generated or analysed during the current study.