.png){kind=icon}
.png){kind=icon}
Return
Language-Driven Multi-Task Manipulation With Action-Mask-Enhanced Multimodal Learning
.png){kind=icon}
PRE
AI
DOI：10.1109/LRA.2026.3674003.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
For long-horizon multi-task robotic manipulation, hierarchical approaches provide an effective way to combine high-level language-based task planning with low-level vision-language based sub-task execution. Then, we propose a framework that integrates a two-stage task planner with a multimodal low-level action planner incorporating an explicit action-mask policy. At high level, a Vision-Language Model (VLM) first perceives object and scene information from observations, and a Large Language Model (LLM) then reasons over this together with a task library and human instruction to generate a textual task plan. This two-stage design mitigates modality bias between perception and planning. At low level, an asymmetric multimodal encoder, SigLIP2 with Weight-Decomposed Low-Rank Adaptation (DoRA) for text and multi-view ResNets for vision, feeds into an Action Chunking with Transformer (ACT)-based policy enhanced by Temperature-Scaled Spatial Attention and Bidirectional Cross-Attention for language-vision fusion. We further introduce an explicit action-mask policy that jointly predicts actions and their validity, enabling real-time sub-task termination detection and robust switching across multiple sub-tasks without additional inference overhead. Experiments on weighing and multi-object manipulation tasks demonstrate performance, with ablation studies validating the contribution of each component. Generalization experiments under out-of-distribution conditions highlight both the strengths and limitations. Deployment on a distinct dual-arm robotic platform in a new scenario validates transferability.
Keywords:
Hierarchical planning
vision-language-action models
imitation learning
bimanual manipulation
Journal
I
IF:
5.3
Papers:
1.7K
Citations:
3.9W
Organization
View more organizations