.png){kind=icon}
返回
Prompt Tuning in Code Intelligence: An Experimental Evaluation
.png){kind=icon}
PRE
AI
DOI：10.1109/TSE.2023.3313881.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Pre-trained models have been shown effective in many code intelligence tasks, such as automatic code summarization and defect prediction. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream task data, while in practice, the data scarcity scenarios are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this article, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with four code intelligence tasks including defect prediction, code search, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all four tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data. We also discuss the implications for adapting prompt tuning in code intelligence tasks.
Keyword:
Tuning
Codes
Task analysis
Training
Predictive models
Adaptation models
Source coding
Code intelligence
prompt tuning
empirical study
期刊
IF:
5.6
论文数:
2.9K
被引数:
1.1W
机构
引用论文
Pth1/Vam3p Is the Syntaxin Homolog at the Vacuolar Membrane of Saccharomyces cerevisae Required for the Delivery of Vacuolar Hydrolases.png)
Genetics
OAAI
GeneticsIF0
OAAI分享收藏Combining dispersal, landscape connectivity and habitat suitability to assess climate-induced changes in the distribution of Cunningham’s skink, Egernia cunninghamiPLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏Treatment of variable valence in organic nomenclature (lambda convention) (Recommendations 1983)OAAI
OAAI分享收藏VulDeeLocator: A Deep Learning-Based Fine-Grained Vulnerability DetectorVulDeeLocator: 一种基于深度学习的细粒度漏洞检测器OAAI
OAAI分享收藏
查看更多内容