Prognostic Value and Reproducibility of AI-assisted Analysis of Lung Involvement in COVID-19 at Low-Dose Submillisievert Chest CT: Sample Size Implications for Clinical Trials
Christopher Gieraerts, Anthony Dangis, Lode Janssen, Annick Demeyere, Yves De Bruecker, Nele De Brucker, Annelies Van Den Bergh, Tine Lauwerier, A. Heremans, Eric Frans, Michaël R. Laurent, Bavo Ector, John Roosen, Annick Smismans, Johan Frans, Marc Gillis, Rolf Symons
Abstract
PURPOSE: To compare the prognostic value and reproducibility of visual versus AI-assisted analysis of lung involvement on submillisievert low-dose chest CT in COVID-19 patients. MATERIALS AND METHODS: This was a HIPAA-compliant, institutional review board-approved retrospective study. From March 15 to June 1, 2020, 250 RT-PCR confirmed COVID-19 patients were studied with low-dose chest CT at admission. Visual and AI-assisted analysis of lung involvement was performed by using a semi-quantitative CT score and a quantitative percentage of lung involvement. Adverse outcome was defined as intensive care unit (ICU) admission or death. Cox regression analysis, Kaplan-Meier curves, and cross-validated receiver operating characteristic curve with area under the curve (AUROC) analysis was performed to compare model performance. Intraclass correlation coefficients (ICCs) and Bland- Altman analysis was used to assess intra- and interreader reproducibility. RESULTS: Adverse outcome occurred in 39 patients (11 deaths, 28 ICU admissions). AUC values from AI-assisted analysis were significantly higher than those from visual analysis for both semi-quantitative CT scores and percentages of lung involvement (all P<0.001). Intrareader and interreader agreement rates were significantly higher for AI-assisted analysis than visual analysis (all ICC ≥0.960 versus ≥0.885). AI-assisted variability for quantitative percentage of lung involvement was 17.2% (coefficient of variation) versus 34.7% for visual analysis. The sample size to detect a 5% change in lung involvement with 90% power and an α error of 0.05 was 250 patients with AI-assisted analysis and 1014 patients with visual analysis. CONCLUSION: AI-assisted analysis of lung involvement on submillisievert low-dose chest CT outperformed conventional visual analysis in predicting outcome in COVID-19 patients while reducing CT variability. Lung involvement on chest CT could be used as a reliable metric in future clinical trials.