.png){kind=icon}
返回
Exploring Flat Minima for Domain Generalization With Large Learning Rates
.png){kind=icon}
OA
AI
DOI：10.1109/TKDE.2024.3392980.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Domain Generalization (DG) aims to generalize to arbitrary unseen domains. A promising approach to improve model generalization in DG is the identification of flat minima. One typical method for this task is SWAD, which involves averaging weights along the training trajectory. However, the success of weight averaging depends on the diversity of weights, which is limited when training with a small learning rate. Instead, we observe that leveraging a large learning rate can simultaneously promote weight diversity and facilitate the identification of flat regions in the loss landscape. However, employing a large learning rate suffers from the convergence problem, which cannot be resolved by simply averaging the training weights. To address this issue, we introduce a training strategy called Lookahead which involves the weight interpolation, instead of average, between fast and slow weights. The fast weight explores the weight space with a large learning rate, which is not converged while the slow weight interpolates with it to ensure the convergence. Besides, weight interpolation also helps identify flat minima by implicitly optimizing the local entropy loss that measures flatness. To further prevent overfitting during training, we propose two variants to regularize the training weight with weighted averaged weight or with accumulated history weight. Taking advantage of this new perspective, our methods achieve state-of-the-art performance on both classification and semantic segmentation domain generalization benchmarks.
Keyword:
Training
Interpolation
Data models
Trajectory
Space exploration
Perturbation methods
Loss measurement
Domain generalization
domain shift
flat minima
large learning rate
期刊
IF:
10.4
论文数:
6.9K
被引数:
3.2W
机构
引用论文
First-Principles Calculations of Acoustic and Anharmonic Properties of Ferromagnetic Cu2MnZ (Z = Al and In) Heusler Alloys铁磁Cu2MnZ (Z = Al和In) Heusler合金的声学和非谐特性的第一性原理计算
.png){kind=icon}
PREAI
分享收藏
分享收藏Do Perceptions of Competence Mediate The Relationship Between Fundamental Motor Skill Proficiency and Physical Activity Levels of Children in Kindergarten?能力的感知是否可以介导幼儿园儿童的基本运动技能熟练程度与身体活动水平之间的关系？
PREAI
分享收藏
查看更多内容