A Cyclical Learning Rate Method in Deep Learning Training
Jiaqi Li, Xiaodong Yang
Abstract
The learning rate is an important hyperparameter for training deep neural networks. The traditional learning rate method has the problems of instability of accuracy. Aiming at these problems, we proposed a new learning rate method with different cyclical changes in each training cycle instead of a fixed value. It achieves higher accuracy in less iterations and faster convergence. Through the experiment on CIFAR-10 and CIFAR-100 datasets based on VGG network and RESNET network, the final results show that the proposed method has better results on stability and accuracy than cyclical learning rate method.
Topics & Concepts
HyperparameterComputer scienceStability (learning theory)Artificial intelligenceArtificial neural networkConvergence (economics)Rate of convergenceDeep learningTraining (meteorology)Machine learningKey (lock)PhysicsComputer securityMeteorologyEconomicsEconomic growthAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningStochastic Gradient Optimization Techniques