PMMNet: A Dual Branch Fusion Network of Point Cloud and Multi-View for Intracranial Aneurysm Classification and Segmentation
Ruifen Cao, Dongwei Zhang, Pi-Jing Wei, Yun Ding, Chun-Hou Zheng, Dayu Tan, Chao Zhou
Abstract
Intracranial aneurysm (IA) is a vascular disease of the brain arteries caused by pathological vascular dilation, which can result in subarachnoid hemorrhage if ruptured. Automatically classification and segmentation of intracranial aneurysms are essential for their diagnosis and treatment. However, the majority of current research is focused on two-dimensional images, ignoring the 3D spatial information that is also critical. In this work, we propose a novel dual-branch fusion network called the Point Cloud and Multi-View Medical Neural Network (PMMNet) for IA classification and segmentation. Specifically, one branch based on 3D point clouds serves the purpose of extracting spatial features, whereas the other branch based on multi-view images acquires 2D pixel features. Ultimately, the two types of features are fused for IA classification and segmentation. To extract both local and global features from 3D point clouds, Multilayer Perceptron (MLP) and the attention mechanism are used in parallel. In addition, a SPSA module is proposed for multi-view image feature learning, which extracts more exquisite channel and spatial multi-scale features from 2D images. Experiments conducted on the IntrA dataset outperform other state-of-the-art methods, demonstrating that the proposed PMMNet exhibits strong superiority on the medical 3D dataset. We also obtain competitive results on public datasets, including ModelNet40, ModelNet10, and ShapeNetPart, which further validate the robustness and generality of the PMMNet.