.png){kind=icon}
返回
SALR: Sharpness-Aware Learning Rate Scheduler for Improved Generalization
.png){kind=icon}
OA
AI
DOI：10.1109/TNNLS.2023.3263393.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In an effort to improve generalization in deep learning and automate the process of learning rate scheduling, we propose SALR: a sharpness-aware learning rate update technique designed to recover flat minimizers. Our method dynamically updates the learning rate of gradient-based optimizers based on the local sharpness of the loss function. This allows optimizers to automatically increase learning rates at sharp valleys to increase the chance of escaping them. We demonstrate the effectiveness of SALR when adopted by various algorithms over a broad range of networks. Our experiments indicate that SALR improves generalization, converges faster, and drives solutions to significantly flatter regions.
Keyword:
Schedules
Deep learning
Neural networks
Convergence
Bayes methods
Training
Stochastic processes
generalization
learning rate schedule
sharpness
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
分享收藏The power of obfuscation techniques in malicious JavaScript code: A measurement study恶意JavaScript代码中混淆技术的功能: 一项测量研究
OAAI
OAAI分享收藏Differentiation of Type 1 ILCs from a Common Progenitor to All Helper-like Innate Lymphoid Cell Lineages.png)
CellOAAI
CellIF0
OAAI分享收藏
查看更多内容