.png){kind=icon}
.png){kind=icon}
Return
An adaptive gradient method with transformation bound
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2025.129369.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Deep learning typically uses optimizers in two categories. The first category contains the stochastic gradient descent (SGD) family, which demonstrates superior generalization performance during the later stages of training. The second category features adaptive methods, such as Adam, which achieve faster convergence in the early stages of training. To ensure effective training, transitioning from adaptive methods to the SGD family after a certain number of iterations is crucial. We propose the AdaTB algorithm, which utilizes a specialized transformation function to enable a smooth transition from Adam to SGDM. Furthermore, we introduce a novel framework and provide a proof of the convergence of AdaTB within this framework in the non-convex setting. Finally, we verify the effectiveness of AdaTB using the Rastrigin function, CIFAR-10/100, and PTB datasets. The code can be accessed at https://github.com/Wenhan-Jiang/AdaTB.git.
Keywords:
Adam
Optimizer
Convergence analysis
Neural networks
Journal
IF:
6.5
Papers:
2.5W
Citations:
6.5W