MixFormer: A Mixed CNN–Transformer Backbone for Medical Image Segmentation

Abstract

Transformers using self-attention mechanisms have recently advanced medical imaging by modeling long-range semantic dependencies, though they lack the ability of convolutional neural networks (CNNs) to capture local spatial details. This study introduced a novel segmentation (SEG) network derived from a mixed CNN-Transformer (MixFormer) feature extraction backbone to enhance medical image segmentation. The MixFormer network seamlessly integrates global and local information from Transformer and CNN architectures during the downsampling process. To comprehensively capture the interscale perspective, we introduced a multiscale spatial-aware fusion (MSAF) module, enabling effective interaction between coarse and fine feature representations. In addition, we proposed a mixed multibranch dilated attention (MMDA) module to bridge the semantic gap between encoding and decoding stages while emphasizing specific regions. Finally, we implemented a CNN-based upsampling approach to recover low-level features, substantially improving segmentation accuracy. Experimental validations on prevalent medical image datasets demonstrated the superior performance of MixFormer. On the Synapse dataset, our approach achieved a mean Dice similarity coefficient (DSC) of 82.64% and a mean Hausdorff distance (HD) of 12.67 mm. On the automated cardiac diagnosis challenge (ACDC) dataset, the DSC was 91.01%. On the international skin imaging collaboration (ISIC) 2018 dataset, the model achieved a mean intersection over union (mIoU) of 0.841, an accuracy of 0.958, a precision of 0.910, a recall of 0.934, and an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${F}1$ </tex-math></inline-formula> score of 0.913. For the Kvasir-SEG dataset, we recorded a mean Dice of 0.9247, an mIoU of 0.8615, a precision of 0.9181, and a recall of 0.9463. On the computer vision center (CVC)-ClinicDB dataset, the results were a mean Dice of 0.9441, an mIoU of 0.8922, a precision of 0.9437, and a recall of 0.9458. These findings underscore the superior segmentation performance of MixFormer compared to most mainstream segmentation networks such as CNNs and other Transformer-based structures.

Topics & Concepts