Transparency assessment of COVID-19 models

Abstract

The COVID-19 pandemic has strained societal structures and created a global crisis. Scientific models have a crucial role in mitigating harm from the pandemic, by estimating the spread of outbreaks of the virus and analysing the effects of public health policies. The context-sensitive and time-sensitive measures provided by COVID-19 models offer real population health impacts and are of great importance. However, these models must be completely transparent before policies and insights are enacted. Transparency is a cornerstone of scientific methodology, and efforts to improve transparency and reproducibility of research have been increasing over the past decade.1McNutt M Journals unite for reproducibility.Science. 2014; 346: 679Crossref PubMed Scopus (202) Google Scholar Researchers have called for complete transparency of COVID-19 models.2Barton CM Alberti M Ames D et al.Call for transparency of COVID-19 models.Science. 2020; 368: 482-483Crossref PubMed Google Scholar An absence of transparency in the design, development, and analysis of these models reduces the trust in their timely messages and limits their reproducibility, impeding scientists from verifying the findings and improving the model's performance. Many modellers have already shared the details of their models openly. However, the overall status of transparency of COVID-19 models remains unknown. We assessed whether COVID-19 modellers adhere to best practices in reporting and documentation; we did not evaluate whether a model's projections are correct. To systematically evaluate the transparency of COVID-19 models, we reviewed a sample of models that have earned global attention and been referenced in governmental public health efforts. We first collected models that included a methods report from the US Centers for Disease Control and Prevention's compilation,3Centers for Disease Control and PreventionCOVID-19 forecasts: deaths.https://www.cdc.gov/coronavirus/2019-ncov/covid-data/forecasting-us.htmlDate: June 26, 2020Date accessed: June 26, 2020Google Scholar then identified the most-referenced models through Google Scholar and the PlumX News Mentions metric. This search took place on June 13, 2020, and resulted in the identification of 29 models. Due to the urgency created by the pandemic, preprints and project websites made available in advance of research publication have had an essential role during the crisis.4Majumder MS Mandl KD Early in the epidemic: impact of preprints on global discourse about COVID-19 transmissibility.Lancet Glob Health. 2020; 8: e627-e630Summary Full Text Full Text PDF PubMed Scopus (122) Google Scholar Therefore, we included models from these sources (n=14), in addition to models in peer-reviewed publications (n=15). We assessed these models against 27 binary criteria to evaluate the transparency of their reports. The transparency assessment criteria were adopted from several transparency checklists,5Hardwicke TE Wallach JD Kidwell MC Bendixen T Crüwell S Ioannidis JPA An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017).R Soc Open Sci. 2020; 7190806Crossref PubMed Scopus (73) Google Scholar, 6Stevens GA Alkema L Black RE et al.Guidelines for Accurate and Transparent Health Estimates Reporting: the GATHER statement.Lancet. 2016; 388: e19-e23Summary Full Text Full Text PDF PubMed Scopus (648) Google Scholar, 7Wallach JD Boyack KW Ioannidis JPA Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017.PLoS Biol. 2018; 16e2006930Crossref PubMed Scopus (148) Google Scholar and include two main themes. First, the specificity of model items, including but not limited to discussion of assumptions, parameterisation, codes, and sensitivity analyses, and second, general research items, such as disclosure of research limitations, funding, and potential conflicts of interest. Two researchers reviewed the full text and appendix of each modelling report. A third reviewer helped to discuss the discrepancies between the first two reviews to reach a resolution. Each of the 27 criteria were satisfied by an average of 22 (76%) of 29 models in our sample. Eight criteria were satisfied by more than 90% of the models, but most criteria were satisfied by a much smaller percentage of models (appendix p 1). For example, seven (24%) of 29 models did not report the equations used, nine (31%) did not report their estimated parameters, 13 (45%) did not share all of their longitudinal data, and 15 (52%) did not report their code (appendix p 2). Only four articles (14%) satisfied more than 90% of our transparency checklist items. This evaluation shows that models that are not fully transparent can still posit analytical insights and inform policy. Rather than presenting recommendations at face value, modellers must ensure that their claims are independently verifiable. The scientific and modelling communities need to make transparency the norm, rather than the exception. Otherwise, they risk losing the faith of policy makers and the public. Such consequences were observed for a model released on March 30, 2020,8Murray CJL Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months.medRxiv. 2020; (published online March 30) (preprint).https://doi.org/10.1101/2020.03.27.20043752Google Scholar which was often cited by government agencies, including the White House.9Shear MD Crowley M Glanz J Coronavirus may kill 100,000 to 240,000 in U.S. despite actions, officials say.https://www.nytimes.com/2020/03/31/us/politics/coronavirus-death-toll-united-states.htmlDate: March 31, 2020Date accessed: June 26, 2020Google Scholar Amid concerns about the model's projections, the scientific community was frustrated by the absence of information regarding codes and specific model details. Another high-profile preprint model published on March 16, 2020,10Ferguson NM Laydon D Nedjati-Gilani G et al.Report 9: impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand.https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdfDate: March 16, 2020Date accessed: June 26, 2020Google Scholar faced similar scrutiny after predicting 510 000 deaths in a scenario with no interventions, prompting researchers to attempt replication. When the code was released about 6 weeks later, several bugs and issues with the modelling assumptions were unearthed. Issues like these suggest that governments should not rely on a small number of models to inform policy. Instead, policy makers can mitigate potential harm by aggregating the available models and synthesising their results to help inform action. A crucial element of model transparency is achieved by providing codes. One concern for researchers is that their code is disorganised and cannot be evaluated or run by laypeople hoping to replicate their efforts. However, even messy code can provide a framework for replication and generate useful dialogue, as seen on platforms like GitHub. Still, well documented code is preferable. Of the 14 (48%) of 29 articles which reported their codes, 13 provided helpful detailed documentation either directly in the file or in supplementary material. We encourage modellers who hope to impact perceptions and policy to follow transparent research practices and release their codes in a timely manner for public evaluation. Many journals ask for transparency statements and encourage scientists to report the details in supplementary documents. Although journals should continue to further enhance their transparency requirements, they cannot control the full transparency of publications. Additionally, peer review cannot be fully relied upon for the most technical aspects of a model, especially if papers do not provide documentation. During crises such as COVID-19, preprints provide fast information delivery. Therefore, journals' transparency policies have minimal effect. Models which were still preprints or project websites satisfied an average of 70% of our transparency criteria, compared with 80% by peer-reviewed articles. The responsibility to provide transparency remains largely with the modellers, even though the peer-review process can help address these omissions. Reporting a fully documented and transparent model can be difficult, but this effort has both tangible and intangible benefits for modellers and policy makers. With the urgency created by a global pandemic, modellers might justify prioritising speed of reporting rather than transparency. However, poor transparency of models that directly impact public health policies (and therefore human lives) can be catastrophic. Model transparency does not necessarily equate to model quality; however, models with little documentation cannot be assessed for quality at all. Hence, all models must be fully transparent for both scientific and ethical purposes. This online publication has been corrected. The corrected version first appeared at thelancet.com/lancetgh on November 18, 2020 This online publication has been corrected. The corrected version first appeared at thelancet.com/lancetgh on November 18, 2020 We declare no competing interests. Download .pdf (.34 MB) Help with pdf files Supplementary appendix Early in the epidemic: impact of preprints on global discourse about COVID-19 transmissibilitySince it was first reported by WHO in Jan 5, 2020, over 80 000 cases of a novel coronavirus disease (COVID-19) have been diagnosed in China, with exportation events to nearly 90 countries, as of March 6, 2020.1 Given the novelty of the causative pathogen (named SARS-CoV-2), scientists have rushed to fill epidemiological, virological, and clinical knowledge gaps—resulting in over 50 new studies about the virus between January 10 and January 30 alone.2 However, in an era where the immediacy of information has become an expectation of decision makers and the general public alike, many of these studies have been shared first in the form of preprint papers—before peer review. Full-Text PDF Open AccessCorrection to Lancet Glob Health 2020; 8: e1459–60Jalali MS, DiGennaro C, Sridhar D. Transparency assessment of COVID-19 models. Lancet Glob Health 2020; 8: e1459–60—In this Correspondence, line two of paragraph seven should have said March 30, and line 11 of paragraph seven should have said March 16. Reference 8 should have been Murray CJL, and reference 10 should have been Ferguson NM, Laydon D, Nedjati-Gilani G, et al. These corrections have been made as of Nov 18, 2020. Full-Text PDF Open AccessThe need for a prediction model assessment framework – Authors' replyWe appreciate the feedback provided by Islam and Khosravi on our assessment.1 We agree that journals play an important role in prioritising transparency and reproducibility in publication decisions. Transparency guidelines have been developed. More than 1100 journal publishers have adopted the Transparency and Openness Promotion guidelines,2 including several journals that published models evaluated in our analysis. Despite these important initiatives and ongoing improvements, journals' peer review processes do not fully assess and control for model transparency. Full-Text PDF Open AccessThe need for a prediction model assessment frameworkWe thank Mohammad Jalali and colleagues (December, 2020)1 for highlighting the need for transparency assessments of COVID-19 models. The authors evaluated the transparency of COVID-19 models using a 27-point binary criterion adopted with the use of three different checklists, and reported that more than half of the studies did not share their longitudinal data, and only 14% of the studies met 90% of the transparency items on their checklist. However, the authors did not consider the MINimum Information for Medical artificial intelligence (AI) Reporting (MINIMAR),2 which provides reporting standards for AI model projections in health care. Full-Text PDF Open Access

Topics & Concepts