.png){kind=icon}
返回
FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair
.png){kind=icon}
OA
AI
DOI：10.1109/TSE.2024.3472476.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Flaky tests are problematic because they non-deterministically pass or fail for the same software version under test, causing confusion and wasting development effort. While machine learning models have been used to predict flakiness and its root causes, there is much less work on providing support to fix the problem. To address this gap, in this paper, we focus on predicting the type of fix that is required to remove flakiness and then repair the test code on that basis. We do this for a subset of flaky tests where the root cause of flakiness is in the test itself and not in the production code. One key idea is to guide the repair process with additional knowledge about the test's flakiness in the form of its predicted fix category. Thus, we first propose a framework that automatically generates labeled datasets for 13 fix categories and trains models to predict the fix category of a flaky test by analyzing the test code only. Our experimental results using code models and few-shot learning show that we can correctly predict most of the fix categories. To show the usefulness of such fix category labels for automatically repairing flakiness, we augment the prompts of GPT 3.5 Turbo, a Large Language Model (LLM), with such extra knowledge to request repair suggestions. The results show that our suggested fix category labels, complemented with in-context learning, significantly enhance the capability of GPT 3.5 Turbo in generating fixes for flaky tests. Based on the execution and analysis of a sample of GPT-repaired flaky tests, we estimate that a large percentage of such repairs, (roughly between 51% and 83%) can be expected to pass. For the failing repaired tests, on average, 16% of the test code needs to be further changed for them to pass.
Keyword:
Codes
Predictive models
Maintenance engineering
Analytical models
Production
Large language models
Java
Few shot learning
Python
Manuals
Flaky tests
fix category
test repair
large language models
code models
few shot learning
software testing
期刊
IF:
5.6
论文数:
2.9K
被引数:
1.1W
机构
引用论文
Isolation and characterization of two types of actinophage infectingStreptomyces scabies感染疥疮链霉菌的两种类型的放线噬菌体的隔离和表征
.png){kind=icon}
PREAI
分享收藏Smells in software test code: A survey of knowledge in industry and academia软件测试代码中的气味: 工业界和学术界的知识调查
PREAI
分享收藏
查看更多内容