.png){kind=icon}
Return
Transformers in source code generation: A comprehensive survey
.png){kind=icon}
PRE
AI
DOI：10.1016/j.sysarc.2024.103193.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Transformers have revolutionized natural language processing (NLP) and have had a huge impact automating tasks. Recently, transformers have led to the development of powerful large language models (LLMs), which have advanced automatic code generation. This study provides a review of code generation concepts and transformer applications in this field. First, the fundamental concepts of the attention mechanism embedded into transformers are explored. Then, predominant automated code generation approaches briefly reviewed, including non -learning code generation (e.g., rule -based), shallow learning (e.g., heuristic rules, grammar -based), and deep learning models. Afterward, this survey reviews pre -training and fine-tuning techniques for code generation, focusing on the application of efficient transformer methods such as parameter efficient tuning, instruction tuning, and prompt tuning. Additionally, this work briefly outlines resources code generation (e.g., datasets, benchmarks, packages) and evaluation metrics utilized in code generation processes. Finally, the challenges and potential research directions (e.g., multimodal learning) are investigated in depth.
Keywords:
Code generation
Transformers
Large language models(LLMs)
Pre-training
Efficient fine-tuning
Prompt-tuning
Alignment-tuning
Journal
IF:
4.1
Papers:
3.0K
Citations:
4.2K
Organization
Cited Papers
Study on Oxygen Control of Large Diameter N-type Monocrystalline Silicon with Large Thermal Field.png)
Silicon
SiliconIF0
PREAI
ShareSaveAutomatic detection of Long Method and God Class code smells through neural source code embeddings
OAAI
OAAIShareSaveMeta-analysis of bariatric surgery versus non-surgical treatment for type 2 diabetes mellitusOncotarget
OAAI
OncotargetIF0
OAAIShareSaveGeochemical evidence for sediment provenance in mudstones and fossil‐poor wackestones (Upper Jurassic, Majorca Island)Terra Nova
Terra NovaIF0
PREAI
ShareSaveShareSaveTreatment of variable valence in organic nomenclature (lambda convention) (Recommendations 1983)OAAI
OAAIShareSave
View more