On the fallibility of simulation models in informing pandemic responses

Abstract

As of April 24, 2020, the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic has led to over 2·7 million confirmed cases, and 190 000 reported deaths worldwide. To reduce the transmission and mortality of this newly emerging infection, countries have needed to make rapid policy decisions on the basis of scarce early data with many uncertainties. In the early stages of pandemics, mathematical models can provide valuable insights into transmission dynamics, help to predict disease spread, and evaluate control measures.1Fitzpatrick MC Bauch CT Townsend JP Galvani AP Modelling microbial infection to address global health challenges.Nat Microbiol. 2019; 4: 1612-1619Crossref PubMed Scopus (30) Google Scholar However, models are only valid within the limits of the parameters examined. As reliable parameter estimates are rarely available early in a new pandemic, best-guess estimates are used, which need to be constantly reviewed as new real-world data emerge. Estimating how sensitive the model is to changes in its parameters can provide useful information about validity when parameters are uncertain. Interpreting models without considering these factors can lead to flawed inferences, which can have far-reaching effects when they inform public health policy. We illustrate the potential impact that flawed model inferences can have on public health policy with the model described in The Lancet Global Health by Joel Hellewell and colleagues,2Hellewell J Abbott S Gimma S et al.Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts.Lancet Glob Health. 2020; 8: e488-e496Summary Full Text Full Text PDF PubMed Scopus (1572) Google Scholar which is part of the scientific evidence informing the UK Government's response to coronavirus disease 2019 (COVID-19).3UK GovernmentScientific Advisory Group for Emergencies (SAGE): coronavirus (COVID-19) response.https://www.gov.uk/government/groups/scientific-advisory-group-for-emergencies-sage-coronavirus-covid-19-responseDate: 2020Date accessed: April 4, 2020Google Scholar On March 12, 2020, the UK Government decided to cease community testing and contact tracing, claiming that the scientific evidence did not support these strategies, as the UK had entered the so-called delay phase of the epidemic. These actions were consistent with the conclusions of Hellewell and colleagues that these measures were unlikely to be able to bring the epidemic under control in under 12 weeks: “In most plausible outbreak scenarios, case isolation and contact tracing alone is insufficient to control outbreaks, and in some scenarios even near perfect contact tracing will still be insufficient, and further interventions would be required to achieve control.”2Hellewell J Abbott S Gimma S et al.Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts.Lancet Glob Health. 2020; 8: e488-e496Summary Full Text Full Text PDF PubMed Scopus (1572) Google Scholar A key parameter of the model was the delay between a case becoming symptomatic and being isolated.2Hellewell J Abbott S Gimma S et al.Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts.Lancet Glob Health. 2020; 8: e488-e496Summary Full Text Full Text PDF PubMed Scopus (1572) Google Scholar Two median delays were modelled—3·83 days (short) and 8·09 days (long)—on the basis of data from the SARS-CoV epidemic, and an empirical distribution calculated from the early phase of the SARS-CoV-2 outbreak in Wuhan.2Hellewell J Abbott S Gimma S et al.Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts.Lancet Glob Health. 2020; 8: e488-e496Summary Full Text Full Text PDF PubMed Scopus (1572) Google Scholar However, by this time, several rapid tests with a turnaround time of less than 4 h had been developed internationally, with some having received relevant regulatory approvals.4Sheridan C Coronavirus and the race to distribute reliable diagnostics.https://www.nature.com/articles/d41587-020-00002-2Date: Feb 19, 2020Date accessed: April 4, 2020Google Scholar Soon after, South Korea and Singapore began drive-through testing and had developed the capacity to test rapidly and deliver results within a single day, making a 1-day delay scenario plausible.5Bedingfield W What the world can learn from South Korea's coronavirus strategy.https://www.wired.co.uk/article/south-korea-coronavirusDate: March 21, 2020Date accessed: April 4, 2020Google Scholar, 6Edward D How South Korea created the world's most ambitious coronavirus testing programme.https://www.itv.com/news/2020-03-31/how-south-korea-created-the-world-s-most-ambitious-coronavirus-testing-programmeDate: March 31, 2020Date accessed: April 4, 2020Google Scholar Indeed, recent models examining interventions for COVID-19 control in Singapore have assumed a 1-day delay between symptom onset and quarantine.7Koo JR Cook AR Park M et al.Interventions to mitigate early spread of SARS-CoV-2 in Singapore: a modelling study.Lancet Infect Dis. 2020; (published online March 23.)https://doi.org/10.1016/S1473-3099(20)30162-6Summary Full Text Full Text PDF PubMed Scopus (491) Google Scholar We find that inferences of the model from Hellewell and colleagues2Hellewell J Abbott S Gimma S et al.Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts.Lancet Glob Health. 2020; 8: e488-e496Summary Full Text Full Text PDF PubMed Scopus (1572) Google Scholar are very sensitive to the parameter of onset-to-isolation delay (appendix). Using the authors' original code, we demonstrate that when the delay is changed to a median of 1 day, the model predicts the probability of controlling the epidemic within 12 weeks to be more than 80%, with 30–60% (ie, considerably less than near perfect) contact tracing (depending on the proportion of pre-symptomatic cases at a given time). These results suggest that rapid testing, contact tracing, and isolation could be effective strategies to control transmission. At the time that the UK Government decided to cease community testing, real-world data from several other countries were pointing to the effectiveness of testing and contact tracing, potentially at odds with the evidence that the government was following. On March 12, 2020 (16 days after the tenth SARS-CoV-2 death in both South Korea and Italy), South Korea had 66 reported deaths compared with Italy's 827 (appendix). The epidemic trajectories in South Korea, Japan, Taiwan, and Singapore have been vastly different from most European countries, with far fewer deaths (appendix). Although multiple aspects of the populations and systems of these countries could have contributed to these differences, one key commonality is the early adoption of an approach focused on testing, isolation, and contact tracing in the countries with slower trajectories. Indeed, many of these countries did not institute nationwide lockdown measures at all or until much later, after the curve had flattened, potentially indicating the impact of case detection-based strategies on transmission (appendix). These detection-based strategies might have allowed the countries to control transmission without stringent lockdown measures and school closures, thus avoiding the disruption that these entail. Instead, they were able to implement targeted restrictions in response to observed outbreaks (eg, church closures in South Korea).8Shim E Tariq A Choi W Lee Y Chowell G Transmission potential and severity of COVID-19 in South Korea.Int J Infect Dis. 2020; (published online March 18.)DOI:10.1016/j.ijid.2020.03.031Summary Full Text Full Text PDF Scopus (456) Google Scholar By contrast, many European countries have had to impose lockdowns much earlier in the timeline of their epidemics (appendix). Empirical, real-world data must be considered alongside mathematical models when devising pandemic responses. Models are fallible and scientists and policy makers must be mindful that an over-reliance on models, and a lack of caution in interpreting them, could be a costly exercise. We declare no competing interests. Download .pdf (.4 MB) Help with pdf files Supplementary appendix Feasibility of controlling COVID-19 outbreaks by isolation of cases and contactsIn most scenarios, highly effective contact tracing and case isolation is enough to control a new outbreak of COVID-19 within 3 months. The probability of control decreases with long delays from symptom onset to isolation, fewer cases ascertained by contact tracing, and increasing transmission before symptoms. This model can be modified to reflect updated transmission characteristics and more specific definitions of outbreak control to assess the potential success of local response efforts. Full-Text PDF Open AccessOn the fallibility of simulation models in informing pandemic responses – Authors' replyWe welcome the correspondence from Deepti Gurdasani and Hisham Ziauddeen and agree that models that influence policy should be open to scrutiny, be informed by the best available evidence, and be constantly reviewed as more information and better estimates emerge. For these very reasons, we made the computer code needed to run the model available with the publication. This allows others to run the model using different parameter values when more information becomes available and explore the resulting effects. Full-Text PDF Open Access

Topics & Concepts