Layer-wise contrastive learning BERT for sentence representation of GitHub

Daoquan Chen

doi:10.1016/j.neucom.2025.131504

Layer-wise contrastive learning BERT for sentence representation of GitHub

2025-09-15

0

PRE

AI

DOI：10.1016/j.neucom.2025.131504

Original

Original request for help

Save

Abstract

En 中文

Every day, on GitHub, end-users submit a large number of issues that must be addressed to ensure the success of software projects. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the sentence representation of [CLS] from the top layer of BERT has a limited ability to capture the semantic meaning of sentences. GitHub issue reports often include code snippets and user-generated terms not found in standard vocabularies. Therefore, the classification predictions of BERT are affected. To generate better sentence semantic representations of BERT for GitHub, we propose a layer-wise Contrastive Learning BERT (CLBERT), which uses contrastive learning to enhance the representation ability by contrasting the layer-by-layer representation. Further, to obtain as comprehensive information as possible, representations of each layer are extracted and learned by an attention mechanism as the final classification features. Finally, experiments conducted on two GitHub data sets show that our proposed model significantly improves classification performance.

Layer-wise contrastive learning BERT for sentence representation of GitHub

Abstract

Journal

Organization