Transfer learning for software vulnerability prediction using Transformer models

Ilias Kalouptsoglou

doi:10.1016/j.jss.2025.112448

Return

Transfer learning for software vulnerability prediction using Transformer models

2025-05-05

PRE

Ilias Kalouptsoglou *

Miltiadis Siavvas

Apostolos Ampatzoglou

Dionysios Kehagias

Alexander Chatzigeorgiou

DOI：10.1016/j.jss.2025.112448

Original

Save

Abstract

En 中文

Recently software security community has exploited text mining and deep learning methods to identify vulnerabilities. To this end, the progress in the field of Natural Language Processing (NLP) has opened a new direction in constructing Vulnerability Prediction (VP) models by employing Transformer-based pre-trained models. This study investigates the capacity of Generative Pre-trained Transformer (GPT), and Bidirectional Encoder Representations from Transformers (BERT) to enhance the VP process by capturing semantic and syntactic information in the source code. Specifically, we examine different ways of using CodeGPT and CodeBERT to build VP models to maximize the benefit of their use for the downstream task of VP. To enhance the performance of the models we explore fine-tuning, word embedding, and sentence embedding extraction methods. We also compare VP models based on Transformers trained on code from scratch or after natural language pre-training. Furthermore, we compare these architectures to state-of-the-art text mining and graphbased approaches. The results showcase that training a separate deep learning predictor with pre-trained word embeddings is a more efficient approach in VP than either fine-tuning or extracting sentence-level features. The findings also highlight the importance of context-aware embeddings in the models' attempt to identify vulnerable patterns in the source code.

Keywords:

Software security

Deep learning

Transfer learning

Transformer

Vulnerability prediction