Evaluating Knowledge Transfer in the Neural Network for Medical Images
Sina Akbarian, Laleh Seyyed-Kalantari, Farzad Khalvati, Elham Dolatabadi
Abstract
The performance of deep learning models, such as convolutional neural networks (CNN)s, is highly dependent on the size of the dataset used for training. In low-data environments, it can be challenging to achieve good performance when training models from scratch. This is where knowledge transfer approaches from pre-trained networks can be particularly useful. In this study, we implement different experiments for standard transfer learning approaches as our baseline and introduce the adoption of a novel knowledge transfer approach, called teacher-student learning framework, to improve the performance of diagnostic predictive models in medical imaging. Specifically, we investigate various configurations in the teacher-student learning framework inspired by the activation attention transfer in computer vision models to help address some of the challenges faced in medical imaging, such as the limited availability of annotated data and limited computing resources. We show that the teacher-student learning approach has the potential to significantly improve the performance of diagnostic predictive models. Our findings could have a positive impact on healthcare accessibility and affordability, as they may enable the development of more cost-effective and widely available medical imaging technologies under a limited data environment.