.png){kind=icon}
返回
Bugs in machine learning-based systems: a faultload benchmark
.png){kind=icon}
PRE
AI
DOI：10.1007/s10664-023-10291-1.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The rapid escalation of applying Machine Learning (ML) in various domains has led to paying more attention to the quality of ML components. There is then a growth of techniques and tools aiming at improving the quality of ML components and integrating them into the ML-based system safely. Although most of these tools use bugs' lifecycle, there is no standard benchmark of bugs to assess their performance, compare them and discuss their advantages and weaknesses. In this study, we firstly investigate the reproducibility and verifiability of the bugs in ML-based systems and show the most important factors in each one. Then, we explore the challenges of generating a benchmark of bugs in ML-based software systems and provide a bug benchmark namely defect4ML that satisfies all criteria of standard benchmark, i.e. relevance, reproducibility, fairness, verifiability, and usability. This faultload benchmark contains 100 bugs reported by ML developers in GitHub and Stack Overflow, using two of the most popular ML frameworks: TensorFlow and Keras. defect4ML also addresses important challenges in Software Reliability Engineering of ML-based software systems, like: 1) fast changes in frameworks, by providing various bugs for different versions of frameworks, 2) code portability, by delivering similar bugs in different ML frameworks, 3) bug reproducibility, by providing fully reproducible bugs with complete information about required dependencies and data, and 4) lack of detailed information on bugs, by presenting links to the bugs' origins. defect4ML can be of interest to ML-based systems practitioners and researchers to assess their testing tools and techniques.
Keyword:
Benchmark
Machine learning-based system
Software bug
Software reliability engineering
Software testing
期刊
IF:
3.6
论文数:
2.0K
被引数:
5.3K
机构
引用论文
Polysialylated Neural Cell Adhesion Molecule Is Involved in Induction of Long-Term Potentiation and Memory Acquisition and Consolidation in a Fear-Conditioning Paradigm聚唾液酸化的神经细胞粘附分子在恐惧调节范式中参与诱导长期增强和记忆获取和巩固
OAAI
.png){kind=icon}
OAAI分享收藏Molecular motion in plastic phases of organometallic compounds: iron-57 Mössbauer and carbon-13 n.m.r. spectra of tricarbonyl(1–5-hapto-cyclohexadienyl)iron tetrafluoroborate and related compounds有机金属化合物的塑料相中的分子运动: 铁-57 m ö ssbauer和碳-13 n.m.r.三羰基 (1-5-半环己二烯基) 四氟硼酸铁和相关化合物的光谱
PREAI
分享收藏
Obstructive sleep apnea classification based on spectrogram patterns in the electrocardiogram基于心电图谱图模式的阻塞性睡眠呼吸暂停分类
PREAI
分享收藏
查看更多内容