BRAU-Net++: U-Shaped Hybrid CNN-Transformer Network for Medical Image Segmentation
Libin Lan, Pengzhou Cai, Lu Jiang, Xiaojuan Liu, Yong Li, Yudong Zhang
Abstract
Accurate medical image segmentation is vital for clinical quantification, disease diagnosis, treatment planning, and other applications. Convolution-based U-shaped architectures excel at learning local features but rely heavily on image-specific inductive biases inherent to convolutions. Transformer-based models, on the other hand, effectively capture long-range dependencies using self-attention but face challenges of quadratic computational and memory demands as sequence lengths increase. To address these limitations, we propose BRAU-Net++, a hybrid CNN-Transformer network that integrates the strengths of both paradigms within a U-shaped architecture. The proposed BRAUNet++ adopts the two key ideas. First, it employs bi-level routing attention as its core building block to hierarchically construct the encoder-decoder structure, enabling efficient learning of global semantics while reducing computational complexity. Second, the network restructures skip connections by incorporating channel-spatial attention, which uses convolution operations to minimize spatial information loss during down-sampling and enhance multi-scale feature interactions. Extensive experiments on four diverse imaging modalities: Synapse, COVID-19, CT-ICH2020, and STS2D2023, demonstrate that BRAU-Net++ outperforms state-of-the-art methods, including its baseline BRAU-Net, under almost all evaluation metrics. These results reveal the model’s generality and robustness for multi-modal medical image segmentation tasks, while also highlighting that BRAU-Net++ achieves an optimal tradeoff between performance and efficiency by integrating dynamic sparse attention with a well-designed U-shaped architecture. Our code is publicly available on GitHub.