Enhancing deep neural network training through learnable adaptive normalization

Abstract

Normalization is a fundamental preprocessing technique in data science, commonly used to standardize data distributions prior to model training. Its primary role is to maintain consistent statistical properties across features, which facilitates efficient learning and enhances training stability. In deep learning, normalization methods are particularly beneficial, as they regulate input distributions within neural networks, promoting more stable training and faster convergence. This study introduces and evaluates a novel approach: learnable adaptive normalization layers integrated into neural networks. Experiments were conducted across nine datasets encompassing feature, image, and time-series data, utilizing various deep learning architectures, including feed-forward, convolutional, and transformer-based neural networks. The results demonstrate that adaptive normalization consistently outperforms traditional methods, such as mean subtraction, standard deviation scaling, and layer normalization. Moreover, in many scenarios, adaptive normalization achieved performance that was either superior to or comparable with batch normalization, with image classification being the notable exception. These findings indicate that adaptive normalization not only accelerates training convergence but also enhances final model performance in most cases, underscoring its effectiveness. Given that the choice of network architecture is inherently a hyperparameter tuning challenge, we recommend considering adaptive normalization in the preprocessing step for future networks. Furthermore, replacing batch or layer normalization with adaptive normalization may lead to improved training efficiency and final performance, depending on the specific problem.

Topics & Concepts