Computational Complexity Reduction Techniques for Deep Neural Networks: A Survey
Md. Bipul Hossain, Na Gong, Mohamed Shaban
Abstract
Deep-learning (DL) has been extensively applied in multiple fields such as medicine, space, agriculture and industry. It has proven to be able to successfully extract unique and discriminative features of targets from structured or unstructured data and provide useful and precise predictions. However, the computational complexity of DL remains a burden for realizing such approaches on battery-powered resource-constrained mobile, wearable, and other Internet-of-Things (IoT) edge devices. In this paper, we present a survey of the significant approaches for reducing the complexity of DL models including pruning, quantization, knowledge distillation, multiplication reduction, resource efficient architectures, low rank approximation and network architecture search. Further, we discuss the limitations of these approaches and demonstrate potential solutions.