.png){kind=icon}
返回
Learning multimodal adaptive relation graph and action boost memory for visual navigation
.png){kind=icon}
PRE
AI
DOI：10.1016/j.aei.2024.102678.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The task of visual navigation (VN) is steering the agent find target object only using visual perceptions. Previous works largely exploit multimodal information (e.g. visual and training memory) to improve the environmental perception ability, while making less effort to leverage interchange information. Besides, multimodal fusion tends to ignore the data dependencies (prefer a part of the modal data) as well as the supervision of the action. In this work, we present a novel multimodal graph learning (MGL) structure for VN, which consists of three parts. (1) the multimodal fusion exploits the rich information across spatial, RGB, and depth information about objects' place, as well as semantic information about their categories, (2) adaptive relation graph (ARG) is dynamically built using object detectors, which encodes multimodal fusion and adapt to a novel environment. It embeds its navigation history and other useful task-oriented structural information, thus make the agent own the association ability and make advisable informed decisions and (3) action boost module (ABM) aims to assist the agent make intelligent decisions, which predicts more accurate action using beneficial training experience. Our agent can foresight what the goal state may look like and how to get closer towards that state. These combinations of the whatand the howallow the agent to navigate to the target object effectively. We validate our approach on the AI2-THOR dataset. It reports 24.2% and 23.7% increase in SPL(Success weighted by Per Length) and SR(Success Rate) compared with baselines, respectively. Code and datasets can be found in https://github.com/luosword/ABM_VN.
Keyword:
Action boost memory
Knowledge graph
Reinforcement learning
Visual navigation
Visual transformer network
期刊
IF:
9.9
论文数:
4.4K
被引数:
1.7W
机构
引用论文
.png){kind=icon}
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations视觉基因组: 使用众包密集图像注释连接语言和视觉
OAAI
OAAI分享收藏分享收藏
Optimal Graph Transformer Viterbi knowledge inference network for more successful visual navigation用于更成功的视觉导航的最佳图形变压器Viterbi知识推理网络
PREAI
分享收藏Cobalt and Copper Composite Oxides as Efficient Catalysts for Preferential Oxidation of CO in H2-Rich Stream钴和铜复合氧化物作为H2-Rich流中CO优先氧化的有效催化剂
PREAI
分享收藏分享收藏
查看更多内容