.png){kind=icon}
Return
Understanding Regression in Code Self-Refinement: A Study of Refinement Protocols Without Execution Feedback
.png){kind=icon}
OA
AI
DOI：10.3390/ai7100416.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Self-refinement can improve generated code without additional training, but without execution feedback, revision can also turn already correct code into incorrect code. We study how self-refinement protocol design affects code correctness using six locally deployable open-weight models, three Python code-generation benchmarks, and 9984 model-task pairs. Using the same initial code for each protocol, we compare direct revision with refinement that adds critique and planning, test whether asking the model if revision is needed before refining affects correctness, and compare separate calls with a single response. Among the evaluated refinement protocols, direct revision achieved the highest overall correctness, whereas adding separate Critique and Planning stages increased repairs but caused substantially more regressions. Direct revision produced 55 repairs and 20 regressions; adding Critique produced 220 repairs and 605 regressions, and adding Planning after Critique produced 206 repairs and 719 regressions. Decision filtering and Single-call organization improved the regression-prone protocols by reducing harmful changes. They improved pass rate by 3.89 to 5.10 and 3.22 to 3.92 percentage points, respectively, but neither approach surpassed direct revision. Repeated direct generation produced many repairs simply by generating another candidate, while direct revision maintained a much more favorable repair-regression balance. These results suggest that, without execution feedback, refinement designs should prioritize preserving already correct code over adding reasoning stages.
Keywords:
large language models
code generation
self-refinement
refinement protocols
correctness regression
Journal
A
IF:
5
Papers:
1.1K
Citations:
941
Organization
Cited Papers
No cited papers available