Multimodal multi-instance evidence fusion neural networks for cancer survival prediction
Hui Luo, Jiashuang Huang, Hengrong Ju, Tianyi Zhou, Weiping Ding
Abstract
Accurate cancer survival prediction plays a crucial role in assisting clinicians in formulating treatment plans. Multimodal data, such as histopathological images, genomic data, and clinical information, provide complementary and comprehensive information, significantly enhancing the accuracy of this task. However, existing methods, despite achieving some promising results, still exhibit two significant limitations: they fail to effectively utilize global context and overlook the uncertainty of different modalities, which may lead to unreliable predictions. In this study, we propose a multimodal multi-instance evidence fusion neural network for cancer survival prediction, called M2EF-NNs. Specifically, to better capture global information from images, we employ a pre-trained vision transformer model to extract patch feature embeddings from histopathological images. Additionally, we are the first to apply the Dempster-Shafer evidence theory to the cancer survival prediction task and introduce subjective logic to estimate the uncertainty of different modalities. We then dynamically adjust the weights of the class probability distribution after multimodal fusion based on the estimated evidence from the fused multimodal data to achieve trusted survival prediction. Finally, the experimental results on three cancer datasets demonstrate that our method significantly improves cancer survival prediction regarding overall C-index and AUC, thereby validating the model's reliability.