Vision Transformers and CNN-Based Knowledge-Distillation for Histopathological Image Classification
Seddik Boudissa, Shyam Sundar Debsarkar, Hiroharu Kawanaka, Bruce J. Aronow, V. B. Surya Prasath
Abstract
Histopathological image analysis remains at the forefront of computational pathology presenting numerous challenges and demanding tasks, primarily due to the complex nature of tissue structures and the extensive scale of whole slide images (WSIs). Deep learning models have been widely used in histopathology image analysis, especially convolutional neural network (CNN)-based models for classification. However, CNNs have certain limitations due to their small receptive field. Recent works employed adaptations of the classical transformer architecture to visual data [1] [2]. Models such as Vision Transformer (ViT) and Swin Transformer leverage the powerful multi-head self-attention mechanism and have demonstrated comparable or superior performance to state-of-the-art CNN-based classification models. Despite their successes, these models require huge amounts of training data to effectively learn representations as they lack the inherent inductive biases of CNNs. This work compares Vision Transformers with baseline CNN models using a breast cancer histopathological dataset. Further, we employ a novel knowledge-distillation approach to enhance the learning efficiency of vits, When trained with a limited amount of data, Unlike previous works, we aimed to minimize convolution operations when generating patch embeddings to preserve spatial information before reaching the transformer attention layers, we achieved an accuracy of 87.7% for the ViT-base trained as a student of ResNet50, which represents a 1.2% improvement in accuracy over the standalone ViT-base. [3]