Context-Aware Robust Fine-Tuning

Xiaofeng Mao

doi:10.1007/s11263-023-01951-2

Context-Aware Robust Fine-Tuning

2023-12-03

3

PRE

AI

DOI：10.1007/s11263-023-01951-2

Original

Original request for help

Save

Abstract

En 中文

Contrastive language-image pre-trained (CLIP) models have zero-shot ability of classifying an image belonging to [CLASS] by using similarity between the image and the prompt sentence a [CONTEXT] of [CLASS]. Based on exhaustive text cues in [CONTEXT], CLIP model is aware of different contexts, e.g. background, style, viewpoint, and exhibits unprecedented robustness against a wide range of distribution shifts. However, recent works find further fine-tuning of CLIPmodels improves accuracy but sacrifices the robustness on downstream tasks. We conduct an empirical investigation to show fine-tuning will corrupt the context-aware ability of pre-trained CLIP features. To solve this problem, we propose Context-Aware Robust Fine-tuning (CAR-FT). CAR-FT regularizes the model during fine-tuning to capture the context information. Specifically, we use zero-shot prompt weights to get the context distribution contained in the image. By minimizing the KullbackLeibler divergence (KLD) between context distributions induced by original/fine-tuned CLIP models, CAR-FT makes the context-aware ability of CLIP inherited into downstream tasks, and achieves both higher in-distribution (ID) and out-ofdistribution (OOD) accuracy. The experimental results show CAR-FT achieves superior robustness on five OOD test datasets of ImageNet, and meanwhile brings accuracy gains on nine downstream tasks. Additionally, CAR-FT surpasses previous domain generalization (DG) methods and gets 78.5% averaged accuracy on DomainBed benchmark, building the new stateof-the-art.

Keywords:

Pre-trained models

CLIP

Fine-tuning

Robustness

Context-Aware Robust Fine-Tuning

Abstract

Journal

Organization