.png){kind=icon}
Return
Automatically Recommend Code Updates: Are We There Yet?
.png){kind=icon}
OA
AI
DOI：10.1145/3678167.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In recent years, large pre-trained Language Models of Code (CodeLMs) have shown promising results on various software engineering tasks. One such task is automatic code update recommendation, which transforms outdated code snippets into their approved and revised counterparts. Although many CodeLM-based approaches have been proposed, claiming high accuracy, their effectiveness and reliability on real-world code update tasks remain questionable. In this article, we present the first extensive evaluation of state-of-the-art CodeLMs for automatically recommending code updates. We assess their performance on two diverse datasets of paired updated methods, considering factors such as temporal evolution, project specificity, method size, and update complexity. Our results reveal that while CodeLMs exhibit higher performance in settings that ignore temporal information, they struggle in more realistic time-wise scenarios and generalize poorly to new projects. Furthermore, CodeLM performance decreases significantly for larger methods and more complex updates. Furthermore, we observe that many CodeLM-generated updates are actually null, especially in time-wise settings, and meaningful edits remain challenging. Our findings highlight the significant gap between the perceived and actual effectiveness of CodeLMs for real-world code update recommendation and emphasize the need for more research on improving their practicality, robustness, and generalizability.
Keywords:
Code updates
neural machine translation
Journal
A
IF:
6.2
Papers:
1.2K
Citations:
3.4K
Organization
Cited Papers
ShareSaveShareSaveMatrix Effects in the Detection of Pb and Ba in Soils Using Laser-Induced Breakdown Spectroscopy
PREAI
ShareSave
Impact of Metformin Use on Lactate Kinetics in Patients with Severe Sepsis and Septic Shock.png)
Shock
ShockIF0
PREAI
ShareSaveFrequency stabilization of a diode laser to Doppler-free spectrum of molecular iodine at 633 nm
PREAI
ShareSave
View more