.png){kind=icon}
返回
Context-Aware Robust Fine-Tuning
.png){kind=icon}
PRE
AI
DOI：10.1007/s11263-023-01951-2.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Contrastive language-image pre-trained (CLIP) models have zero-shot ability of classifying an image belonging to [CLASS] by using similarity between the image and the prompt sentence a [CONTEXT] of [CLASS]. Based on exhaustive text cues in [CONTEXT], CLIP model is aware of different contexts, e.g. background, style, viewpoint, and exhibits unprecedented robustness against a wide range of distribution shifts. However, recent works find further fine-tuning of CLIPmodels improves accuracy but sacrifices the robustness on downstream tasks. We conduct an empirical investigation to show fine-tuning will corrupt the context-aware ability of pre-trained CLIP features. To solve this problem, we propose Context-Aware Robust Fine-tuning (CAR-FT). CAR-FT regularizes the model during fine-tuning to capture the context information. Specifically, we use zero-shot prompt weights to get the context distribution contained in the image. By minimizing the KullbackLeibler divergence (KLD) between context distributions induced by original/fine-tuned CLIP models, CAR-FT makes the context-aware ability of CLIP inherited into downstream tasks, and achieves both higher in-distribution (ID) and out-ofdistribution (OOD) accuracy. The experimental results show CAR-FT achieves superior robustness on five OOD test datasets of ImageNet, and meanwhile brings accuracy gains on nine downstream tasks. Additionally, CAR-FT surpasses previous domain generalization (DG) methods and gets 78.5% averaged accuracy on DomainBed benchmark, building the new stateof-the-art.
Keyword:
Pre-trained models
CLIP
Fine-tuning
Robustness
期刊
IF:
9.3
论文数:
3.9K
被引数:
2.8W
机构
引用论文
.png){kind=icon}
分享收藏
分享收藏Do Perceptions of Competence Mediate The Relationship Between Fundamental Motor Skill Proficiency and Physical Activity Levels of Children in Kindergarten?能力的感知是否可以介导幼儿园儿童的基本运动技能熟练程度与身体活动水平之间的关系？
PREAI
分享收藏
查看更多内容