.png){kind=icon}
返回
Automatically Recommend Code Updates: Are We There Yet?
.png){kind=icon}
OA
AI
DOI：10.1145/3678167.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In recent years, large pre-trained Language Models of Code (CodeLMs) have shown promising results on various software engineering tasks. One such task is automatic code update recommendation, which transforms outdated code snippets into their approved and revised counterparts. Although many CodeLM-based approaches have been proposed, claiming high accuracy, their effectiveness and reliability on real-world code update tasks remain questionable. In this article, we present the first extensive evaluation of state-of-the-art CodeLMs for automatically recommending code updates. We assess their performance on two diverse datasets of paired updated methods, considering factors such as temporal evolution, project specificity, method size, and update complexity. Our results reveal that while CodeLMs exhibit higher performance in settings that ignore temporal information, they struggle in more realistic time-wise scenarios and generalize poorly to new projects. Furthermore, CodeLM performance decreases significantly for larger methods and more complex updates. Furthermore, we observe that many CodeLM-generated updates are actually null, especially in time-wise settings, and meaningful edits remain challenging. Our findings highlight the significant gap between the perceived and actual effectiveness of CodeLMs for real-world code update recommendation and emphasize the need for more research on improving their practicality, robustness, and generalizability.
Keyword:
Code updates
neural machine translation
期刊
A
IF:
6.2
论文数:
1.2K
被引数:
3.4K
机构
引用论文
分享收藏Matrix Effects in the Detection of Pb and Ba in Soils Using Laser-Induced Breakdown Spectroscopy使用激光诱导击穿光谱检测土壤中Pb和Ba的基质效应
PREAI
分享收藏Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues精炼ChatGPT生成的代码: 描述和缓解代码质量问题OAAI
OAAI分享收藏
Impact of Metformin Use on Lactate Kinetics in Patients with Severe Sepsis and Septic Shock.png)
Shock
ShockIF0
PREAI
分享收藏Frequency stabilization of a diode laser to Doppler-free spectrum of molecular iodine at 633 nm
PREAI
分享收藏
查看更多内容