Transformers in source code generation: A comprehensive survey

doi:10.1016/j.sysarc.2024.103193

Transformers in source code generation: A comprehensive survey

2024-08-01

1

PRE

AI

DOI：10.1016/j.sysarc.2024.103193

Original

Original request for help

Save

Abstract

En 中文

Transformers have revolutionized natural language processing (NLP) and have had a huge impact automating tasks. Recently, transformers have led to the development of powerful large language models (LLMs), which have advanced automatic code generation. This study provides a review of code generation concepts and transformer applications in this field. First, the fundamental concepts of the attention mechanism embedded into transformers are explored. Then, predominant automated code generation approaches briefly reviewed, including non -learning code generation (e.g., rule -based), shallow learning (e.g., heuristic rules, grammar -based), and deep learning models. Afterward, this survey reviews pre -training and fine-tuning techniques for code generation, focusing on the application of efficient transformer methods such as parameter efficient tuning, instruction tuning, and prompt tuning. Additionally, this work briefly outlines resources code generation (e.g., datasets, benchmarks, packages) and evaluation metrics utilized in code generation processes. Finally, the challenges and potential research directions (e.g., multimodal learning) are investigated in depth.

Keywords:

Code generation

Transformers

Large language models(LLMs)

Pre-training

Efficient fine-tuning

Prompt-tuning

Alignment-tuning

Transformers in source code generation: A comprehensive survey

Abstract

Journal

Organization