.png){kind=icon}
返回
Measuring code efficiency optimization capabilities with ACEOB
.png){kind=icon}
OA
AI
DOI：10.1016/j.jss.2024.112250.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
As Moore's Law gains diminish, software performance and efficiency become increasingly vital. Optimizing code efficiency is challenging, even for professional programmers. However, related research remains relatively scarce, and rigorously assessing models' abilities to optimize code efficiency is fraught with difficulties. In response to this challenge, we first conduct an in-depth analysis of code patternsin the model training dataset, meticulously exploring human-written code. Secondly, we define a task for optimizing code efficiency and introduce the A utomatic C ode E fficiency O ptimization B enchmark (ACEOB), which consists of 95,359 pairs of efficient-inefficient code aimed at assessing code efficiency optimization capabilities. To our knowledge, ACEOB is the first dataset specifically targeting Python code efficiency optimization. To evaluate models' ability in optimizing code efficiency, we propose two new metrics: the I somorphic O ptimal C omparison C ode B LEU (IOCCB) metric and the N ormalized P erformance I ndex (NPI) metric, to assess the efficiency of model-generated code. We also evaluate several advanced code models, such as PolyCoder and CodeT5, after fine-tuning them on ACEOB and demonstrate that the efficiency of each model improves after introducing the NPI filter. However, it was observed that even ChatGPT does not perform optimally in code efficiency optimization tasks. Our dataset and models are available at: https://github.com/CodeGeneration2/ACEOB.
Keyword:
Coding efficiency optimization
Benchmark datasets
Code generation
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
4.1
论文数:
5.4K
被引数:
8.4K
机构
引用论文
.png){kind=icon}
Temperature monitoring options available at the Idaho national laboratory advanced test reactor在爱达荷国家实验室先进试验反应堆可用的温度监测选项OAAI
OAAI分享收藏Hydrothermal preparation and low temperature magnetic properties of TbOOH, DyOOH, HoOOH, ErOOH, and YbOOHTbOOH，DyOOH，HoOOH，ErOOH和YbOOH的水热制备和低温磁性
PREAI
分享收藏Controlled partial embedding of carbon nanotubes within flexible transparent layers碳纳米管在柔性透明层内的受控部分嵌入
PREAI
分享收藏
查看更多内容