Improving the Performance of Neural Networks with an Ensemble of Activation Functions
Arijit Nandi, Nanda Dulal Jana, Swagatam Das
Abstract
Activation functions in the neural networks play an important role by introducing non-linear properties to the neural networks. Thus it is considered as one of the essential ingredients among other building blocks of a neural network. But the selection of the appropriate activation function for the enhancement of model accuracy is strenuous in a sense; the performance of the NN-model is influenced by a proper selection of activation function for a dataset. Proper activation function selection is still a trial and error method for which the model accuracy improves for classification. As a solution to this problem, we have proposed an activation function ensembling by majority voting that has significantly improved the model accuracy in a classification context. The proposed model is tested on four benchmark datasets such as MNIST, Fashion MNIST, Semeion, and ARDIS IV datasets. The result shows that the performance of the proposed model is appreciably better than other traditional methods such as Convolutional Neural Network (CNN), Support Vector Machine (SVM), Recurrent Neural Network (RNN).