Design and Implementation of an Approximate Softmax Layer for Deep Neural Networks
Yue Gao, Weiqiang Liu, Fabrizio Lombardi
Abstract
Deep neural networks (DNNs) have been widely used in classification due to their high accuracy. The softmax function is one of the important non-linear functions in DNNs. Therefore, high performance and efficient hardware design are sought. However, the improvement of the softmax function is difficult because the exponent and the division units are complex. In this paper, we propose new approximate hardware architectures for both the exponent and the division units. Compared with the state-of-the-art designs, the proposed approximate softmax design consumes significantly less resources and also achieves high performance while maintaining a very high accuracy.
Topics & Concepts
Softmax functionDivision (mathematics)Computer scienceExponentArtificial neural networkFunction (biology)Deep neural networksLayer (electronics)Artificial intelligencePattern recognition (psychology)AlgorithmArithmeticMathematicsChemistryEvolutionary biologyLinguisticsPhilosophyBiologyOrganic chemistryAdvanced Neural Network ApplicationsParallel Computing and Optimization TechniquesNeural Networks and Applications