.png){kind=icon}
返回
Bugs in large language models generated code: an empirical study
.png){kind=icon}
OA
AI
DOI：10.1007/s10664-025-10614-4.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Large Language Models (LLMs) for code have gained significant attention recently. They can generate code in different programming languages based on provided prompts, fulfilling a long-lasting dream in Software Engineering (SE), i.e., automatic code generation. Similar to human-written code, LLM-generated code is prone to bugs, and these bugs have not yet been thoroughly examined by the community. Given the increasing adoption of LLM-based code generation tools (e.g., GitHub Copilot) in SE activities, it is critical to understand the characteristics of bugs contained in code generated by LLMs. This paper examines samples of 333 bugs collected from code generated using three leading LLMs (i.e., CodeGen, PanGu-Coder, and Codex) and identifies the following 10 distinctive bug patterns: Misinterpretations, Syntax Error, Silly Mistake, Prompt-biased code, Missing Corner Case, Wrong Input Type, Hallucinated Object, Wrong Attribute, Incomplete Generation, and Non-Prompted Consideration. The bug patterns are presented in the form of a taxonomy. The identified bug patterns are validated using online surveys with over 50 LLM practitioners and researchers. The surveyed participants generally asserted the significance and prevalence of the bug patterns. Researchers and practitioners can leverage these findings to develop effective quality assurance techniques for LLM-generated code. This study sheds light on the distinctive characteristics of LLM-generated code.
Keyword:
Large language models
Bugs
Software testing
Empirical study
期刊
IF:
3.6
论文数:
2.0K
被引数:
5.3K
机构
引用论文
Serum lipid abnormalities in heart transplant recipients: predominance of HDL2-like particles in the HDL pattern心脏移植受者的血脂异常: HDL模式中HDL2-like颗粒的优势
.png){kind=icon}
PREAI
分享收藏
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues精炼ChatGPT生成的代码: 描述和缓解代码质量问题OAAI
OAAI分享收藏Silent bugs in deep learning frameworks: an empirical study of Keras and TensorFlow深度学习框架中的沉默bug: 基于Keras和TensorFlow的实证研究
PREAI
分享收藏分享收藏
查看更多内容