Deep Learning Systems for Pneumothorax Detection on Chest Radiographs: A Multicenter External Validation Study
Yee Liang Thian, Dianwen Ng, James Thomas Patrick Decourcy Hallinan, Pooja Jagmohan, Soon Yiew Sia, Cher Heng Tan, Yong Han Ting, Pin Lin Kei, Geoiphy George Pulickal, Vincent Tze Yang Tiong, Swee Tian Quek, Mengling Feng
Abstract
Purpose To assess the generalizability of a deep learning pneumothorax detection model on datasets from multiple external institutions and examine patient and acquisition factors that might influence performance. Materials and Methods In this retrospective study, a deep learning model was trained for pneumothorax detection by merging two large open-source chest radiograph datasets: ChestX-ray14 and CheXpert. It was then tested on six external datasets from multiple independent institutions (labeled A–F) in a retrospective case-control design (data acquired between 2016 and 2019 from institutions A–E; institution F consisted of data from the MIMIC–CXR dataset). Performance on each dataset was evaluated by using area under the receiver operating characteristic curve (AUC) analysis, sensitivity, specificity, and positive and negative predictive values, with two radiologists in consensus being used as the reference standard. Patient and acquisition factors that influenced performance were analyzed. Results The AUCs for pneumothorax detection for external institutions A–F were 0.91 (95% CI: 0.88, 0.94), 0.97 (95% CI: 0.94, 0.99), 0.91 (95% CI: 0.85, 0.97), 0.98 (95% CI: 0.96, 1.0), 0.97 (95% CI: 0.95, 0.99), and 0.92 (95% CI: 0.90, 0.95), respectively, compared with the internal test AUC of 0.93 (95% CI: 0.92, 0.93). The model had lower performance for small compared with large pneumothoraces (AUC, 0.88 [95% CI: 0.85, 0.91] vs AUC, 0.96 [95% CI: 0.95, 0.97]; P = .005). Model performance was not different when a chest tube was present or absent on the radiographs (AUC, 0.95 [95% CI: 0.92, 0.97] vs AUC, 0.94 [95% CI: 0.92, 0.05]; P > .99). Conclusion A deep learning model trained with a large volume of data on the task of pneumothorax detection was able to generalize well to multiple external datasets with patient demographics and technical parameters independent of the training data. Keywords: Thorax, Computer Applications-Detection/Diagnosis See also commentary by Jacobson and Krupinski in this issue. Supplemental material is available for this article. ©RSNA, 2021