.png){kind=icon}
.png){kind=icon}
Return
AdaDB: An adaptive gradient method with data-dependent bound
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2020.07.070.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Adaptive gradient learning methods such as Adam, RMSProp, and AdaGrad play an essential role in train-ing a very deep neural network. The learning rates of these optimizers are adaptively changed to accelerate the training process. And the convergence speed is much faster than SGD in many deep learning tasks such as classification and NLP tasks. However, recent works have pointed out that adaptive learning methods can not converge to a critical point under some situations and suffer poor generalization in many deep learning tasks. In our study, we propose AdaDB, an adaptive learning optimizer with data dependent bound on the learning rate. Every element in the learning rate vector is constrained between a dynamic upper bound and a constant lower bound. And the upper bound is dependent on the data. We also give a theoretical proof of the convergence of AdaDB in the non-convex setting. Our experiments show that AdaDB is capable of eliminating the generalization gap between Adam and SGD. Experiments also reveal the convergence speed of AdaDB is much faster than Adam. (c) 2020 Elsevier B.V. All rights reserved.
Keywords:
Optimizer
Adaptive gradient descent
Generalization
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.