.png){kind=icon}
Return
Evaluating LLMs in ROS robotic software code generation
.png){kind=icon}
PRE
AI
DOI：10.1007/s10664-026-10910-7.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Code generation has emerged as a valuable technique for accelerating robotic software development. However, unlike general-purpose programming, generating code for Cyber-Physical Systems (CPS) requires navigating complex, domain-specific frameworks like the Robot Operating System (ROS). Although Large Language Models (LLMs) excel in general-purpose code generation, their capability to synthesize executable, middleware-compliant ROS code remains poorly understood. This task demands that LLMs not only solve robotic tasks but also generate code that seamlessly integrates within the distributed ROS framework. To address this gap, we construct ROSDevEval, a specialized benchmark comprising 240 real-world ROS programming tasks, and conduct a comprehensive empirical study to evaluate four state-of-the-art LLMs alongside a specialized code assistant (GitHub Copilot). Our results reveal a severe domain capability gap: while state-of-the-art LLMs achieve up to 88.4% Pass@1 on general-purpose C++, their performance suffers a precipitous relative decline ranging from 78.8% to 85.4% in robotic contexts, and even the specialized assistant struggles to exceed a 32.5% success rate. Through a rigorous open-coding analysis, we systematically categorize LLMs’ generation failures into two distinct dimensions. In robotic task resolution, LLMs struggle to apply physical constraints and produce functional logic. In ROS framework integration, they systematically violate strict middleware contracts required for system orchestration. Furthermore, our root cause analysis explicitly decouples these surface-level failures from their underlying etiology, revealing a profound dual cognitive deficit: LLMs lack the long-horizon procedural decomposition capability to translate abstract tasks into executable sequences, and they fail to grasp the rigid architectural topology required for distributed system integration. These findings delineate current limitations and provide actionable, architecture-aware implications for advancing the next generation of robotic foundation models. Code and datasets are available at https://github.com/zyx065/rosdeveval .
Keywords:
Code generation
Robot software
Robot operating system
Large language model
Journal
IF:
3.6
Papers:
2.0K
Citations:
5.3K
Organization
Cited Papers
.png){kind=icon}
ShareSaveShareSaveShareSave
View more