Comparative analysis of activation functions in neural networks

Firuz Kamalov, Amril Nazir, Murodbek Safaraliev, Aswani Kumar Cherukuri, Rita Zgheib

20212021 28th IEEE International Conference on Electronics, Circuits, and Systems (ICECS)23 citationsDOI

Abstract

Although the impact of activations on the accuracy of neural networks has been covered in the literature, there is little discussion about the relationship between the activations and the geometry of neural network model. In this paper, we examine the effects of various activation functions on the geometry of the model within the feature space. In particular, we investigate the relationship between the activations in the hidden and output layers, the geometry of the trained neural network model, and the model performance. We present visualizations of the trained neural network models to help researchers better understand and intuit the effects of activation functions on the models.

Topics & Concepts

Artificial neural networkComputer scienceActivation functionFeature (linguistics)Artificial intelligenceNervous system network modelsSpace (punctuation)Neural activityNetwork modelTime delay neural networkMachine learningTypes of artificial neural networksNeurosciencePsychologyLinguisticsOperating systemPhilosophyNeural Networks and ApplicationsModel Reduction and Neural NetworksDomain Adaptation and Few-Shot Learning