A Brief Review of the Most Recent Activation Functions for Neural Networks
Marina Adriana Mercioni, Ştefan Holban
Abstract
Even though the majority of the current functions have shortcomings, this study looks at a few activation function capabilities that might lead to performance enhancement. The research on neural networks still shows a lot of interest in the activation function since it can enhance performance. With or without trainable parameters, other adaptive activation functions have been put forth that have demonstrated to lead to better results than the benchmark. These studies outline the characteristics, benefits, constraints, and directions of such types of applications. Due to their shortcomings, several of those functions are now regarded as deprecated. The primary emphasis of such functions is on fundamental elements that are thought to be necessary for learning, such as monocity, derivatives, and finite of their range. The goal of this research article is to present and assess the most popular and recent activation functions. This will go through their characteristics, advantages and disadvantages, formulation, and usage.