DermViT: Diagnosis-Guided Vision Transformer for Robust and Efficient Skin Lesion Classification
Xuejun Zhang, Yehui Liu, Ganxin Ouyang, Wenkang Chen, Ao‐Bo Xu, Takeshi Hara, Xiangrong Zhou, Dongbo Wu
Abstract
Early diagnosis of skin cancer can significantly improve patient survival. Currently, skin lesion classification faces challenges such as lesion-background semantic entanglement, high intra-class variability, artifactual interference, and more, while existing classification models lack modeling of physicians' diagnostic paradigms. To this end, we propose DermViT, a medically driven deep learning architecture that addresses the above issues through a medically-inspired modular design. DermViT consists of three main modules: (1) Dermoscopic Context Pyramid (DCP), which mimics the multi-scale observation process of pathological diagnosis to adapt to the high intraclass variability of lesions such as melanoma, then extract stable and consistent data at different scales; (2) Dermoscopic Hierarchical Attention (DHA), which can reduce computational complexity while realizing intelligent focusing on lesion areas through a coarse screening-fine inspection mechanism; (3). Dermoscopic Feature Gate (DFG), which simulates the observation-verification operation of doctors through a convolutional gating mechanism and effectively suppresses semantic leakage of artifact regions. Our experimental results show that DermViT significantly outperforms existing methods in terms of classification accuracy (86.12%, a 7.8% improvement over ViT-Base) and number of parameters (40% less than ViT-Base) on the ISIC2018 and ISIC2019 datasets. Our visualization results further validate DermViT's ability to locate lesions under interference conditions. By introducing a modular design that mimics a physician's observation mode, DermViT achieves more logical feature extraction and decision-making processes for medical diagnosis, providing an efficient and reliable solution for dermoscopic image analysis.