Building of Computationally Effective Deep Learning Models using Attention-Guided Knowledge Distillation
V A Ashwath, A. Ayyagari, C R Deebakkarthi, R. Arun
Abstract
CNNs and Edge computing have proliferated in the last decade. While these technologies are stupendous in their own right, new horizons can be opened up if they were to be combined. Deep and complex CNNs, which are ones that produce groundbreaking accuracies, are not feasible for deployment in a resource constraint environment. The vanguard in model compression is an innovative technique known as Knowledge Distillation. This paper proposes a way to perform knowledge distillation by means of Attention Maps. The generation of attention maps from features is achieved through the use of ATCam. Not only does this method effectively distills knowledge it also nullifies another flaw of CNNs-black-boxiness. This may lead to leapfrogs in Mobile Machine Aided Medical Diagnosis by deploying interpretable complex CNN in small devices. Three ways to combine the feature maps were experimented with in this research work, which are Feature Map Accumulation(FMA), Maximum Intensity Projection(MIPs) and Saliency Maps. The performance of the proposed approaches are tested on PADUFES-20 dataset and all the three approaches provided significant boost in the performance over the traditional transfer learning methods. The best of these methods - saliency maps - gains a accuracy boost of 9.56% over the generic methods of knowledge transfer. Moreover, to one’s surprise, the resultant model not only matches the teacher model in accuracy, it exceeds it by 1.52% while being only 43.2% of original teacher model’s size.