A Decreasing Scaling Transition Scheme from Adam to SGD
Kun Zeng, Jinlan Liu, Zhixia Jiang, Dongpo Xu
Abstract
Abstract Adaptive gradient algorithm and its variants, such as RMSProp, Adam, AMSGrad, etc., have been widely used in deep learning. Although these algorithms are faster in the early phase of training, their generalization performance is often not as good as stochastic gradient descent (SGD). Hence, a trade‐off method of transforming Adam to SGD after a certain iteration to gain the merits of both algorithms is theoretically and practically significant. To that end, a decreasing scaling transition scheme to achieve a smooth and stable transition from Adam to SGD, which is called DSTAdam. The convergence of the proposed DSTAdam is also proved in an online convex setting. Finally, the effectiveness of the DSTAdam is verified on the different datasets. The implementation is available at: https://github.com/kunzeng/DSTAdam .