.png){kind=icon}
.png){kind=icon}
Return
Self-supervised learning with ensemble representations
.png){kind=icon}
PRE
AI
DOI：10.1016/j.engappai.2025.110007.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Many computer vision applications, such as medical image processing, have struggled with a lack of labeled data. Recently, contrastive self-supervised learning has made remarkable progress in unsupervised representation learning and become a promising alternative in situations with limited labeled data. The simple Siamese network (SimSiam) is a well-known example in this area, known for its simplicity despite its powerful performance. However, it is known to be sensitive to changes in training configurations, such as hyperparameters and augmentation settings, due to its structural characteristics. To address this issue, we focus on the structural similarity between contrastive learning and the teacher-student framework in knowledge distillation. Inspired by the ensemble-based knowledge distillation approach, we propose a new self-supervised learning method, the ensemble representation model for simple Siamese networks (EnSiam), by introducing an ensemble representation in pseudo labels. This can reduce the variance in the contrastive learning procedure, providing better performance. Experiments demonstrate that EnSiam performs better than existing methods for popular datasets, showing that EnSiam is capable of learning high-quality representations. Furthermore, we extend our approach to various other self-supervised learning methods and empirically confirm that ensemble representations can consistently improve performance in general self-supervised learning.
Keywords:
Self-supervised learning
Contrastive learning
Ensemble representations
Representation learning
Variance reduction
Journal
IF:
8
Papers:
5.3K
Citations:
3.5W