$\log$-Sigmoid Activation-Based Long Short-Term Memory for Time-Series Data Classification
Priyesh Ranjan, Pritam Khan, Sudhir Kumar, Sajal K. Das
Abstract
With the enhanced usage of artificial-intelligence-driven applications, the researchers often face challenges in improving the accuracy of data classification models, while trading off the complexity. In this article, we address the classification of time-series data using the long short-term memory (LSTM) network while focusing on the activation functions. While the existing activation functions, such as sigmoid and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\tanh$</tex-math></inline-formula> , are used as LSTM internal activations, the customizability of these activations stays limited. This motivates us to propose a new family of activation functions, called <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\log$</tex-math></inline-formula> -sigmoid, inside the LSTM cell for time-series data classification and analyze its properties. We also present the use of a linear transformation (e.g., <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\log \tanh$</tex-math></inline-formula> ) of the proposed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\log$</tex-math></inline-formula> -sigmoid activation as a replacement of the traditional <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\tanh$</tex-math></inline-formula> function in the LSTM cell. Both the cell activation and recurrent activation functions inside the LSTM cell are modified with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\log$</tex-math></inline-formula> -sigmoid activation family while tuning the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\log$</tex-math></inline-formula> bases. Furthermore, we report a comparative performance analysis of the LSTM model using the proposed and the state-of-the-art activation functions on multiple public time-series databases.