An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer
Yuan Gao, Sofía Ventura‐Díaz, Xin Wang, Muzhen He, Zeyan Xu, Arlene Weir, Hong-Yu Zhou, Tianyu Zhang, Frederieke van Duijnhoven, Luyi Han, Xiao‐Mei Li, Anna D’Angelo, Valentina Laurita Longo, Zaiyi Liu, Jonas Teuwen, Marleen Kok, Regina G. H. Beets‐Tan, Hugo M. Horlings, Tao Tan, Ritse M. Mann
Abstract
Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP’s clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness. Deep learning for medical image analysis is a promising new avenue to predict treatment response, however the clinical application of these methods has been so far limited. Here, the authors propose a model to predict chemotherapy response in breast cancer in real world clinical settings.