Activated Gradients for Deep Neural Networks

Mei Liu, Liangming Chen, Xiaohao Du, Long Jin, Mingsheng Shang

2021IEEE Transactions on Neural Networks and Learning Systems267 citationsDOI

Abstract

Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this article, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this article gives conditions that the GAF needs to meet and, on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this article proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.

Topics & Concepts

Deep neural networksArtificial neural networkComputer sciencePascal (unit)Saddle pointArtificial intelligenceConvergence (economics)MathematicsGeometryProgramming languageEconomic growthEconomicsAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and ELM