An end-to-end content-aware generative adversarial network based method for multimodal medical image fusion
Manisha Das, Deep Gupta, Ashwini M. Bakde
Abstract
The fusion of multimodality medical images combines the most relevant information of the source modalities to improve diagnostic accuracy. Most recently, end-to-end deep learning (DL) based image fusion approaches have shown more accurate and robust fusion performance by combining the feature extraction, selection, and fusion steps together. However, the DL model must learn to extract and preserve distinct features from source images, each of which portrays distinct aspects of the underlying tissues. In this paper, an end-to-end multimodal medical image fusion method is presented using a content-aware generative adversarial network based approach. During the training phase, the generator is trained to generate fake fused images from the source multimodal image pairs. Two discriminators are used to differentiate between the source and fake fused images. Further, the loss function is made content-aware to elevate the preservation of the characteristic information of each of the source images. During the testing phase, the discriminators are discarded and the learned generator model is used to generate the fused images from a multimodal image pair. The subjective and quantitative performance analysis of the extensive experimentation demonstrates that the proposed method achieves notable improvement compared to recent DL based image fusion methods.