Classification of Breast Cancer in Mri with Multimodal Fusion
Margarida Morais, Francisco Maria Calisto, Carlos Santiago, Clara Aleluia, Jacinto C. Nascimento
Abstract
Magnetic resonance imaging (MRI) is the recommended imaging modality in the diagnosis of breast cancer. However, each MRI scan comprises dozens of volumes for the radiologist to inspect, each providing its own set of information on the tissues being scanned. This paper proposes a multimodal framework that processes all the available MRI data in order to reach a diagnosis, instead of relying on a single volume, mimicking the radiologists’ workflow. The framework comprises a 3D convolutional neural network for each modality, whose predictions are then combined using a late fusion strategy based on Dempster-Shafer theory. Results highlight the most relevant modalities required to obtain accurate diagnosis, in agreement with clinical practice. They also show that combining multiple modalities leads to better overall results than their individual counterparts.