.png){kind=icon}
返回
Adaptive hierarchical hyper-gradient descent
.png){kind=icon}
OA
AI
DOI：10.1007/s13042-022-01625-4.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Adaptive learning rate strategies can lead to faster convergence and better performance for deep learning models. There are some widely known human-designed adaptive optimizers such as Adam and RMSProp, gradient based adaptive methods such as hyper-descent and practical loss-based stepsize adaptation (L4), and meta learning approaches including learning to learn. However, the existing studies did not take into account the hierarchical structures of deep neural networks in designing the adaptation strategies. Meanwhile, the issue of balancing adaptiveness and convergence is still an open question to be answered. In this study, we investigate novel adaptive learning rate strategies at different levels based on the hyper-gradient descent framework and propose a method that adaptively learns the optimizer parameters by combining adaptive information at different levels. In addition, we show the relationship between regularizing over-parameterized learning rates and building combinations of adaptive learning rates at different levels. Moreover, two heuristics are introduced to guarantee the convergence of the proposed optimizers. The experiments on several network architectures, including feed-forward networks, LeNet-5 and ResNet-18/34, show that the proposed multi-level adaptive approach can significantly outperform many baseline adaptive methods in a variety of circumstances.
Keyword:
Deep learning
Hypergradient descent
Learning rate adaptation
Hierarchical learning rate system
Adabound
期刊
IF:
2.7
论文数:
3.2K
被引数:
5.6K
机构
引用论文
Metal to Insulator Transition in Epitaxial Graphene Induced by Molecular Doping分子掺杂诱导外延石墨烯的金属-绝缘体转变OAAI
OAAI分享收藏
Gradient-based learning applied to document recognition基于梯度的学习在文档识别中的应用.png)
PROCEEDINGS OF THE IEEE
PROCEEDINGS OF THE IEEEIF25.9
PREAI
分享收藏分享收藏
查看更多内容