Digital Implementation of the Softmax Activation Function and the Inverse Softmax Function
S Raghuram, Anirudh S Bharadwaj, S K Deepika, Mridula S Khadabadi, Aditya Jayaprakash
Abstract
An increase in interest in Deep Neural Networks can be attributed to the recent successes of Deep Learning in various AI applications. Deep Neural Networks form the implementation platform for all these application domains. The next level of adoption is through dedicated hardware implementations of these models, for example in edge-based applications. If a Deep Neural Network is used to represent a classification problem, the last layer is typically the Softmax activation function. Due to the appearance of the exponential function in these implementations, additional effort must be made to realize a digital implementation. In this work, two activation functions-the Softmax and the Inverse Softmax function-as well as the digital implementations of each are explored for their effectiveness in performance and power consumption. The CORDIC technique is used to model the exponential functions in this paper. The Inverse Softmax function, proposed in this paper for the first time, avoid the requirement of the division operator in the Softmax function. Through experiments it has been shown that this function leads to an optimized implementation, as compared to the Softmax activation function.