.png){kind=icon}
返回
Boosting sharpness-aware training with dynamic neighborhood
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patcog.2024.110496.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Learning algorithms motivated by minimizing the sharpness of loss surface is a hot research topic in improving generalization. The existing methods usually solve a constrained min-max problem to minimize sharpness and find flat minima. However, most constraints (i.e., the neighborhood of the sharpness) are inappropriate, leading to sub -optimal results. This paper theoretically explores the optimal neighborhood from the view of Probably Approximately Correct-Bayesian (PAC-Bayesian) framework. A closed form of the optimal neighborhood is provided. This neighborhood is determined by the Hessian matrix and the scales of parameters. Then a generalization bound is derived that serves as a guiding principle in the design of the sharpness minimization algorithm. The Dynamic neighborhood -based Sharpness -Aware Minimization algorithm is proposed, which can adaptively adjust the neighborhood during the training process to gain better performance. Also, the root algorithm is proved can convergent at the rate O (log T / T ) . Experimental results demonstrate that the proposed algorithm outperforms the other methods (e.g., accuracy + 2.86% over baseline on CIFAR-100 for VGG-16).
Keyword:
Flat minima
Generalization
Optimization
Sharpness-aware minimization
期刊
IF:
7.6
论文数:
1.3W
被引数:
4.5W
机构
引用论文
Visualising basins of attraction for the cross-entropy and the squared error neural network loss functions.png)
NEUROCOMPUTING
OAAI
NEUROCOMPUTINGIF6.5
OAAI分享收藏Synthesis of n-type semiconducting diamond film using diphosphorus pentaoxide as the doping source以五氧化二磷为掺杂源合成n型半导体金刚石膜
PREAI
分享收藏Optimization of Texture-Modified Yellowfin Sole (Pleuronectes aspera) by Enzymatic Treatment and Superheated Steam Treating to Improve Quality CharacteristicsProcessesOAAI
ProcessesIF0
OAAI分享收藏Evaluation of the Approach for the Identification of Trajectory Anomalies on CCTV Video from Road IntersectionsMathematics
OAAI
MathematicsIF0
OAAI分享收藏Testing the concurrent validity of a naturalistic upper extremity reaching task测试自然主义上肢到达任务的同时有效性OAAI
OAAI分享收藏Developing Wind and/or Solar Powered Crop Irrigation Systems for the Great Plains为大平原开发风能和/或太阳能作物灌溉系统
PREAI
分享收藏
查看更多内容