.png){kind=icon}
.png){kind=icon}
Return
AI failures in the eyes of the downstream developer: a first look at concerns, practices, and challenges
.png){kind=icon}
OA
AI
H
M
W
H
J
C
DOI：10.1007/s10664-026-10937-w.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
With the advancement of AI models, more software systems are adopting AI as a component to facilitate automation. Pre-trained models (PTMs) have become a cornerstone of AI-based software, allowing for rapid integration and development with lower training cost. However, their adoption also introduces failure modes such as data leakage and biased outputs, that may require careful handling by downstream developers. While previous research has proposed taxonomies of these technical concerns and various mitigation strategies, how downstream developers address these issues during the development of general AI-based software when reusing PTMs remains unexplored. Understanding downstream developers’ perspectives is essential because they directly influence how these potential failure concerns translate into practice, such as determining whether immediate risks like data leakage or model bias are recognised, mitigated, or inadvertently overlooked in real-world deployments. This study investigates downstream developers’ concerns, practices and perceived challenges regarding practical AI failures during the development of AI-based software. To achieve this, we conducted a mixed-method study, including interviews with 16 participants, a survey of 86 practitioners, and an analysis of 874 AI incidents from the AI Incident Database. Our results reveal that while developers generally demonstrate strong awareness of potential AI failures, their practices, especially during the preparation and model selection phases, are often inadequate. The lack of concrete guidelines and policies leads to significant variability in the comprehensiveness of their approaches throughout the development lifecycle, with additional challenges such as poor documentation and knowledge gaps, further impeding effective implementation. Based on our findings, we offer suggestions for AI model contributors, developers of AI-based software, researchers, and policy makers to enhance the integration of failure mitigation measures aimed at mitigating direct harms from AI failures.
Keywords:
AI-based Software
AI failure
Pre-trained models
Mixed methods
Empirical software engineering
Journal
IF:
3.6
Papers:
1.9K
Citations:
5.3K
Organization
Cited Papers