.png){kind=icon}
.png){kind=icon}
Return
MinEnt: Minimum entropy for self-supervised representation learning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patcog.2023.109364.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Self-supervised representation learning is becoming more , more popular due to its superior perfor-mance. According to the information entropy theory, the smaller the information entropy of a feature, the more certain it is and the less redundant it is. Based on this, we propose a simple yet effective self-supervised representation learning method via Minimum Entropy (MinEnt). From the perspective of reducing information entropy, our MinEnt takes the output of the projector towards its nearest minimum entropy as the optimization target. The core of our MinEnt consists of three important steps: 1) normal-ize along the batch dimension to avoid model collapse, 2) compute the nearest minimum entropy to get the target, 3) compute the loss and backpropagate to optimize the network. Our MinEnt can learn ef-ficient representations, even without the need for techniques such as negative sample pairs, predictors, momentum encoders, cross-correlation matrices, etc. Experimental results on four widely used datasets show that our method achieves competitive results in a simple manner.(c) 2023 Elsevier Ltd. All rights reserved.
Keywords:
Self-supervised learning
Minimum entropy
Unsupervised representation learning
Image classification