.png){kind=icon}
Return
Reassessing One-Round Test-Time Refinement for Code Generation
.png){kind=icon}
OA
AI
DOI：10.3390/ai7090353.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Test-time refinement aims to improve generated programs through additional inference, but its value after an initial candidate has been produced remains unclear. We conduct a controlled evaluation of one-round Self-Refine and Self-Debug across seven models and three Python code-generation benchmarks. For each model and task, both methods refine the same initial candidate, allowing us to measure refinement gain without variation in initial generation. Self-Debug produces positive refinement gain in 16 of the 21 model–benchmark combinations and no change in the remaining five, whereas Self-Refine reduces correctness in 16 combinations and improves it in only four. Decomposing refinement gain into repairs and regressions clarifies this contrast. After a diagnostic pass, Self-Debug’s candidate preservation leaves 34.1% of initial final-test failures unaddressed, but 99.95% of initially correct candidates remain correct. With regressions nearly absent, repairs after diagnostic failures translate directly into positive gain. Self-Refine also repairs initial failures, but its regression count is more than three times its repair count overall, producing predominantly negative gain. Resource analysis shows that Self-Refine uses more tokens while generally reducing correctness, whereas Self-Debug provides a more favorable gain–overhead balance, although its inference overhead per net additional pass varies across models and benchmarks. These results show that one-round refinement is not inherently beneficial and should be applied only when its expected gain justifies the additional computation and cost.
Keywords:
large language models
code generation
test-time refinement
self-refinement
self-debugging
execution feedback
Journal
A
IF:
5
Papers:
1.0K
Citations:
941
Organization
Cited Papers
.png){kind=icon}
ShareSaveShareSaveShareSaveShareSave
View more