Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge
Maximilian Zenk, Ujjwal Baid, Sarthak Pati, Akis Linardos, Brandon Edwards, Micah Sheller, Patrick Foley, Alejandro Aristizábal, David Zimmerer, А. Д. Груздев, Jason Martin, Russell T. Shinohara, Annika Reinke, Fabian Isensee, Santhosh Parampottupadam, Kaushal Parekh, Ralf Floca, Hasan Kassem, Bhakti Baheti, Siddhesh Thakur, Verena Chung, Kaisar Kushibar, Karim Lekadir, Meirui Jiang, Youtan Yin, Hongzheng Yang, Quande Liu, Cheng Chen, Qi Dou, Pheng‐Ann Heng, Xiaofan Zhang, Shaoting Zhang, Muhammad Irfan Khan, Mohammad Ayyaz Azeem, Mojtaba Jafaritadi, Esa Alhoniemi, Elina Kontio, Suleiman A. Khan, Leon Mächler, Ivan Ezhov, Florian Kofler, Suprosanna Shit, Johannes C. Paetzold, Timo Loehr, Benedikt Wiestler, Himashi Peiris, Kamlesh Pawar, Shenjun Zhong, Zhaolin Chen, Munawar Hayat, Gary F. Egan, Mehrtash Harandi, Ece Isik-Polat, Görkem Polat, Altan Koçyiğit, Alptekin Temizel, Anup Tuladhar, Lakshay Tyagi, Raissa Souza, Nils D. Forkert, Pauline Mouchès, Matthias Wilms, Vishruth Shambhat, Akansh Maurya, Shubham Subhas Danannavar, Rohit Kalla, Vikas Kumar Anand, Ganapathy Krishnamurthi, Sahil Nalawade, Chandan Ganesh, Benjamin Wagner, Divya Reddy, Yudhajit Das, Fang Yu, Baowei Fei, Ananth J. Madhuranthakam, Joseph A. Maldjian, Gaurav Singh, Jianxun Ren, Wei Zhang, Ning An, Qingyu Hu, Youjia Zhang, Ying Zhou, Vasilis Siomos, Giacomo Tarroni, Jonathan Passerat‐Palmbach, Ambrish Rawat, Giulio Zizzo, Swanand Kadhe, Jonathan P. Epperlein, Stefano Braghin, Yuan Wang, Renuga Kanagavelu, Qingsong Wei, Yechao Yang, Yong Liu, Krzysztof Kotowski, Szymon Adamski, Bartosz Machura
Abstract
Computational competitions are the standard for benchmarking medical image analysis algorithms, but they typically use small curated test datasets acquired at a few centers, leaving a gap to the reality of diverse multicentric patient data. To this end, the Federated Tumor Segmentation (FeTS) Challenge represents the paradigm for real-world algorithmic performance evaluation. The FeTS challenge is a competition to benchmark (i) federated learning aggregation algorithms and (ii) state-of-the-art segmentation algorithms, across multiple international sites. Weight aggregation and client selection techniques were compared using a multicentric brain tumor dataset in realistic federated learning simulations, yielding benefits for adaptive weight aggregation, and efficiency gains through client sampling. Quantitative performance evaluation of state-of-the-art segmentation algorithms on data distributed internationally across 32 institutions yielded good generalization on average, albeit the worst-case performance revealed data-specific modes of failure. Similar multi-site setups can help validate the real-world utility of healthcare AI algorithms in the future.