Initialization of ReLUs for Dynamical Isometry
Rebekka Burkholz, Alina Dubatovka
Abstract
Deep learning relies on good initialization schemes and hyperparameter choices \nprior to training a neural network. Random weight initializations induce random \nnetwork ensembles, which give rise to the trainability, training speed, and sometimes also generalization ability of an instance. In addition, such ensembles provide \ntheoretical insights into the space of candidate models of which one is selected \nduring training. The results obtained so far rely on mean field approximations \nthat assume infinite layer width and that study average squared signals. We derive \nthe joint signal output distribution exactly, without mean field assumptions, for \nfully-connected networks with Gaussian weights and biases, and analyze deviations \nfrom the mean field results. For rectified linear units, we further discuss limitations \nof the standard initialization scheme, such as its lack of dynamical isometry, and \npropose a simple alternative that overcomes these by initial parameter sharing.