Optimization on multifractal loss landscapes explains a diverse range of geometrical and dynamical properties of deep learning
Andrew Ly, Pulin Gong
Abstract
Gradient descent and its variants are foundational in solving optimization problems across many disciplines. In deep learning, these optimizers demonstrate a remarkable ability to dynamically navigate complex loss landscapes, ultimately converging to solutions that generalize well. To elucidate the mechanism underlying this ability, we introduce a theoretical framework that models the complexities of loss landscapes as multifractal. Our model unifies and explains a broad range of realistic geometrical signatures of loss landscapes, including clustered degenerate minima, multiscale structure, and rich optimization dynamics in deep neural networks, such as the edge of stability, non-stationary anomalous diffusion, and the extended edge of chaos without requiring fine-tuning parameters. We further develop a fractional diffusion theory to illustrate how these optimization dynamics, coupled with multifractal structure, effectively guide optimizers toward smooth solution spaces housing flatter minima, thus enhancing generalization. Our findings suggest that the complexities of loss landscapes do not hinder optimization; rather, they facilitate the process. This perspective not only has important implications for understanding deep learning but also extends potential applicability to other disciplines where optimization unfolds on complex landscapes.