Automatically Recommend Code Updates: Are We There Yet?

doi:10.1145/3678167

Automatically Recommend Code Updates: Are We There Yet?

2024-12-03

0

OA

AI

Y

Yue Liu

C

Chakkrit Tantithamthavorn *

Y

Yonghui Liu

P

Patanamon Thongtanunam

L

Li Li

DOI：10.1145/3678167

Original

Original request for help

Save

Abstract

En 中文

In recent years, large pre-trained Language Models of Code (CodeLMs) have shown promising results on various software engineering tasks. One such task is automatic code update recommendation, which transforms outdated code snippets into their approved and revised counterparts. Although many CodeLM-based approaches have been proposed, claiming high accuracy, their effectiveness and reliability on real-world code update tasks remain questionable. In this article, we present the first extensive evaluation of state-of-the-art CodeLMs for automatically recommending code updates. We assess their performance on two diverse datasets of paired updated methods, considering factors such as temporal evolution, project specificity, method size, and update complexity. Our results reveal that while CodeLMs exhibit higher performance in settings that ignore temporal information, they struggle in more realistic time-wise scenarios and generalize poorly to new projects. Furthermore, CodeLM performance decreases significantly for larger methods and more complex updates. Furthermore, we observe that many CodeLM-generated updates are actually null, especially in time-wise settings, and meaningful edits remain challenging. Our findings highlight the significant gap between the perceived and actual effectiveness of CodeLMs for real-world code update recommendation and emphasize the need for more research on improving their practicality, robustness, and generalizability.

Keywords:

Code updates

neural machine translation

Automatically Recommend Code Updates: Are We There Yet?

Abstract

Journal

Organization