Measurement Methodology: Visualizing Uncertainty in Machine Learning-Assisted Measurements
Shervin Shirmohammadi
Abstract
From the onset of the COVID-19 pandemic, many researchers rushed to design Machine Learning (ML)-assisted diagnostic tools that could, supposedly, detect COVID-19 fast and reliably. ML seemed perfect for this job since we had access to many COVID-19 datasets, so a datadriven approach should have quickly yielded such diagnostic tools that could then be distributed to the masses. Unfortunately, the reality fell way short of the expectations. In an extensive study, Wynants and colleagues screened 126,978 relevant titles in the literature and found 412 studies describing 731 such ML-based COVID-19 diagnostic tools, but their conclusion was that “most published prediction model studies were poorly reported and at high risk of bias such that their reported predictive performances are probably optimistic” [1]. Only 29 models had low risk of bias and “should be validated before clinical implementation.” This was confirmed by another study that identified 2,212 such tools, of which 415 were included after initial screening, and 62 were systematically reviewed. The result? “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases” [2]. There were several problems with the proposed tools, but the one that relates to our article is summarized in the following remedial recommendation of the authors: “When reporting results, it is important to include confidence intervals to reflect the uncertainty in the estimate, especially when training models on the small sample sizes commonly seen with COVID-19 data.”