High-Precision Method and Architecture for Base-2 Softmax Function in DNN Training
Yuan Zhang, Lele Peng, Lianghua Quan, Zhang Yong-gang, Shubin Zheng, Hui Chen
Abstract
Softmax is a common and complex activation function in Deep Neural Networks (DNN). However, it is a challenge to apply it efficiently in DNN training hardware accelerator. Therefore, we propose a high precision calculation method and architecture based on base-2 softmax, which has low hardware complexity than base- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$e$ </tex-math></inline-formula> softmax but can still be useful in DNN training. First, we simplify the hardware implementation complexity of calculating base-2 softmax. Second, we use the base-2 hyperbolic COordinate Rotation Digital Computer (CORDIC) to implement the core computation. Finally, we show that the proposed method can be used in DNN training through experiments. Moreover, with the same order of the magnitude of high precision, our hardware cost is lower than traditional base- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$e$ </tex-math></inline-formula> softmax or other alternative design methods. Under TSMC 28nm CMOS technology, an example design of our architecture has the area of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$98787.43\mu m^{2}$ </tex-math></inline-formula> and the power consumption of 24.72mW for circuit synthesis at the frequency of 1GHz.