Deep contrastive representation learning for supervised tasks

Chenguang Duan

doi:10.1016/j.patcog.2024.111309

Return

Deep contrastive representation learning for supervised tasks

2025-05-01

PRE

DOI：10.1016/j.patcog.2024.111309

Original

Original request for help

Save

Abstract

En 中文

Representation learning, with its capability to extract and encapsulate the inherent structure and information embedded within raw data, assumes a crucial role in facilitating the establishment of models. In this paper, we adopt a deep contrastive learning paradigm to derive representations of raw data, thereby laying the foundation for an exploration into supervised learning that encompasses both regression and classification problems. This methodological approach is delineated as a two-stage process. In the initial stage, data representations are acquired through contrastive learning, employing deep ReLU neural networks. Subsequently, in the second stage, the obtained representations are harnessed to address regression and classification in downstream tasks, utilizing norm-constrained ReLU neural networks. Theoretically, we present an informative property and rigorously establish error bounds for the excess risk associated with the resulting estimators of the two-stage method by balancing the stochastic and approximation errors. This establishment serves as a guarantee for the efficacy of deep contrastive learning, especially in scenarios where the sample size during the pretraining phase exceeds that allocated for downstream tasks. Numerical experiments corroborate the effectiveness of the proposed two-stage method and demonstrate a notable alignment with the theoretical findings.

Keywords:

Contrastive learning

Deep neural networks

Error bound

Supervised learning