Temperature-free loss function for contrastive learning

Kim, Bum Jun

doi:10.1016/j.neunet.2026.109222

Temperature-free loss function for contrastive learning

2026-09-29

3

PRE

AI

DOI：10.1016/j.neunet.2026.109222

原文链接

原文求助

摘要

En 中文

As one of the most promising methods in self-supervised learning, contrastive learning has achieved a series of breakthroughs across numerous fields. A predominant approach to implementing contrastive learning is applying InfoNCE loss: By capturing the similarities between pairs, InfoNCE loss enables learning the representation of data. Albeit its success, adopting InfoNCE loss requires tuning a temperature, which is a core hyperparameter for calibrating similarity scores. Despite its significance and sensitivity to performance being emphasized by several studies, searching for a valid temperature requires extensive trial-and-error-based experiments, which increases the difficulty of adopting InfoNCE loss. To address this difficulty, we propose a novel method to deploy InfoNCE loss without temperature. Specifically, we replace temperature scaling with the inverse hyperbolic tangent function, resulting in a modified InfoNCE loss. In addition to hyperparameter-free deployment, we observed that the proposed method even yielded a performance gain in contrastive learning. Our detailed theoretical analysis discovers that the current practice of temperature scaling in InfoNCE loss causes serious problems in gradient descent, whereas our method provides desirable gradient properties. The proposed method was validated on five benchmarks on contrastive learning, yielding satisfactory results without temperature tuning.

Keyword:

InfoNCE loss

Self-supervised learning

Contrastive learning

Temperature

Gradient descent optimization

Hyperparameter tuning

Temperature-free loss function for contrastive learning

摘要

期刊

机构

引用论文