Dynamic bound adaptive gradient methods with belief in observed gradients

Abstract

In the realm of deep neural network training, both AdaBound and AdaBelief have emerged as significant advancements within adaptive gradient methods. AdaBelief, by incorporating curvature information through the belief in observed gradients, effectively addresses the issue of noisy gradients, thereby enhancing optimization stability. However, we found that AdaBelief exhibits sensitivity to abrupt gradient outliers due to over-smoothed second-moment estimation and lacks convergence guarantees in convex settings caused by non-monotonic stepsize sequences. To address these challenges, we introduce AdaBoB (AdaBound with Belief in Observed Gradients), an innovative optimization algorithm that integrates the adaptive learning rate bounds of AdaBound with the curvature-informed belief mechanism of AdaBelief. By combining these two approaches, AdaBoB leverages the strengths of both methods to achieve robust convergence and generalization. Specifically, the curvature-informed belief mechanism mitigates the impact of noisy gradients, while the dynamic bounds ensure that the stepsize remains within a theoretically sound range, thereby satisfying convergence conditions while suppressing outlier-induced instability. We provide a comprehensive theoretical analysis that establishes AdaBoB’s convergence in both convex and non-convex optimization scenarios. Additionally, extensive comparative experiments demonstrate that AdaBoB outperforms state-of-the-art methods across various deep learning tasks. The code is released at https://gitee.com/frontxiang/torch_adabob .

Topics & Concepts