Boosting source code suggestion with self-supervised Transformer Gated Highway

doi:10.1016/j.jss.2022.111553

Boosting source code suggestion with self-supervised Transformer Gated Highway

2023-02-01

7

PRE

AI

DOI：10.1016/j.jss.2022.111553

Original

Original request for help

Save

Abstract

En 中文

Attention-based transformer language models have shown significant performance gains in various natural language tasks. In this work, we explore the impact of transformer language models on the task of source code suggestion. The core intention of this work is to boost the modeling performance for the source code suggestion task and to explore how the training procedures and model architectures impact modeling performance. Additionally, we propose a transformer-based self-supervised learning technique called Transformer Gated Highway that outperforms recurrent and transformer language models of comparable size. The proposed approach combines the Transformer language model with Gated Highway introducing a notion of recurrence. We compare the performance of the proposed approach with transformer-based BERT (CodeTran), RoBERTa (RoBERTaCode), GPT2 (TravTrans), CodeGen and recurrent neural language-based LSTM (CodeLSTM) models. Moreover, we have experimented with various architectural settings for the transformer models to evaluate their impact on modeling performance. The extensive evaluation of the presented approach exhibits better performance on two programming language datasets; Java and C#. Additionally, we have adopted the presented approach for the syntax error correction task to predict the correct syntax token to render its possible implications for other source code modeling tasks.(c) 2022 Elsevier Inc. All rights reserved.

Keywords:

Deep learning

Transformer models

Source code modeling

Source code suggestion

Boosting source code suggestion with self-supervised Transformer Gated Highway

Abstract

Journal

Organization