.png){kind=icon}
返回
Boosting source code suggestion with self-supervised Transformer Gated Highway
.png){kind=icon}
PRE
AI
DOI：10.1016/j.jss.2022.111553.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Attention-based transformer language models have shown significant performance gains in various natural language tasks. In this work, we explore the impact of transformer language models on the task of source code suggestion. The core intention of this work is to boost the modeling performance for the source code suggestion task and to explore how the training procedures and model architectures impact modeling performance. Additionally, we propose a transformer-based self-supervised learning technique called Transformer Gated Highway that outperforms recurrent and transformer language models of comparable size. The proposed approach combines the Transformer language model with Gated Highway introducing a notion of recurrence. We compare the performance of the proposed approach with transformer-based BERT (CodeTran), RoBERTa (RoBERTaCode), GPT2 (TravTrans), CodeGen and recurrent neural language-based LSTM (CodeLSTM) models. Moreover, we have experimented with various architectural settings for the transformer models to evaluate their impact on modeling performance. The extensive evaluation of the presented approach exhibits better performance on two programming language datasets; Java and C#. Additionally, we have adopted the presented approach for the syntax error correction task to predict the correct syntax token to render its possible implications for other source code modeling tasks.(c) 2022 Elsevier Inc. All rights reserved.
Keyword:
Deep learning
Transformer models
Source code modeling
Source code suggestion
期刊
IF:
4.1
论文数:
5.5K
被引数:
8.4K
机构
引用论文
.png){kind=icon}
分享收藏Exemplar: A Source Code Search Engine for Finding Highly Relevant ApplicationsExemplar: 用于查找高度相关应用程序的源代码搜索引擎
OAAI
OAAI分享收藏CodeGRU: Context-aware deep learning with gated recurrent unit for source code modelingCodeGRU: 具有用于源代码建模的门控循环单元的上下文感知深度学习
PREAI
分享收藏分享收藏
查看更多内容