Systematic comparison of deep-learning based fusion strategies for multi-modal ultrasound in diagnosis of liver cancer
Ming‐De Li, Wei Li, Manxia Lin, Xin-Xin Lin, Hang-Tong Hu, Ying-Chen Wang, Si‐Min Ruan, Ze-Rong Huang, Rui-Fang Lu, LV Li, Ming Kuang, Ming-De Lu, Li‐Da Chen, Wei Wang, Qinghua Huang, Qinghua Huang
Abstract
For the diagnosis of liver cancer, conventional brightness mode (B-mode) can only provide morphological information. Multi-modal ultrasound, including shear-wave elastography (SWE) and contrast enhanced ultrasound (CEUS), can provide comprehensive diagnostic information on tumor microenvironment and tissue perfusion . The challenge is to effectively explore the multi-modal features of ultrasound. Besides, there are many fusion strategies currently available, but there is a lack of systematic comparative research on the various fusion strategies. In this study, we designed 'Lesions Pairing' to construct the dataset, addressing the challenge of small sample sizes in multi-modal learning. We then compared the effectiveness of different strategies and proposed hybrid-fusion strategies based on the combination of conventional layer-level fusion (i.e. early-fusion, mid-fusion and late-fusion), which can efficiently extract intra-/inter- modal information. Specifically, we first systematically compared different deep-learning-based fusion strategies for multi-modal ultrasound in the diagnosis of liver cancer. Secondly, based on the comparison results of a multimodal framework that integrates B-mode, SWE, CEUS ultrasound data, and clinical data simultaneously, we propose a hybrid-fusion strategies for the diagnosis of hepatocellular carcinoma and intrahepatic cholangiocarcinoma . The experimental results showed that the area under the curve of the early-late fusion strategy combined with clinical data was 0.9854, which was superior to other single mode and other fusion strategies, increasing by 13.8–25.88 % and 2.22 %-9.79 %, respectively.