.png){kind=icon}
Return
Investigating Self-Supervised Methods for Label-Efficient Learning
.png){kind=icon}
OA
AI
DOI：10.1007/s11263-025-02397-4.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Vision transformers combined with self-supervised learning have enabled the development of models which scale across large datasets for several downstream tasks, including classification, segmentation, and detection. However, the potential of these models for low-shot learning across several downstream tasks remains largely under explored. In this work, we conduct a systematic examination of different self-supervised pretext tasks, namely contrastive learning, clustering, and masked image modelling, to assess their low-shot capabilities by comparing different pretrained models. In addition, we explore the impact of various collapse avoidance techniques, such as centring, ME-MAX, and sinkhorn, on these downstream tasks. Based on our detailed analysis, we introduce a framework that combines mask image modelling and clustering as pretext tasks. This framework demonstrates superior performance across all examined low-shot downstream tasks, including multi-class classification, multi-label classification and semantic segmentation. Furthermore, when testing the model on large-scale datasets, we show performance gains in various tasks.
Keywords:
Vision transformers
Self-supervised learning
Low-shot learning
Masked image modelling
Journal
IF:
9.3
Papers:
4.0K
Citations:
2.8W
Organization
Cited Papers
.png){kind=icon}
ShareSave
ShareSaveShareSave
View more