.png){kind=icon}
返回
InterREC: An Interpretable Method for Referring Expression Comprehension
.png){kind=icon}
PRE
AI
DOI：10.1109/TMM.2023.3251111.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Referring Expression Comprehension (REC) aims to locate the target object in the image according to a referring expression. This is a challenging task owing to the need for understanding both natural language and visual information and interpretable reasoning between them. Most existing implicit reasoning-based REC methods lack interpretability, while explicit reasoning-based REC methods have lower accuracy. To achieve competitive accuracy while providing adequate interpretability, in this work, we propose a novel explicit reasoning-based method named InterREC. First, in order to address the challenge of multi-modal understanding, we design two neural network modules based on text-image representation learning: a Text-Region Matching Module to align objects in the image and noun phrases in the expression, and a Text-Relation Matching Module to align relations between objects in the image and relational phrases in the expression. Additionally, we design a Reasoning Order Tree for handling complex expressions, which can reduce complex expressions to multiple object-relation-object triplets and therefore identify the inference order and reduce the difficulty of reasoning. At the same time, to achieve an interpretable reasoning step, we design a Bayesian Network-based explicit reasoning method. Based on the comparative evaluation on various datasets, our method achieves higher accuracy than existing explicit reasoning-based REC methods, and the visualization results demonstrate the method's high interpretability.
Keyword:
Cognition
Visualization
Task analysis
Transformers
Feature extraction
Linguistics
Representation learning
Referring expression comprehension
transfer learning
Bayesian network
reasoning
期刊
IF:
9.7
论文数:
4.6K
被引数:
2.4W
机构
暂无机构信息
引用论文
分享收藏An improved iterative approach with a comprehensive friction model for identifying dynamic parameters of collaborative robots.png)
Robotica
RoboticaIF0
PREAI
分享收藏Substrate specificity and inhibitor analyses of human steroid 5β-reductase (AKR1D1)人类类固醇5 β-还原酶 (AKR1D1) 的底物特异性和抑制剂分析Steroids
OAAI
SteroidsIF0
OAAI分享收藏A retrospective 5-year review of rubella in South Africa prior to the introduction of a rubella-containing vaccinePLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏Communication and reproductive behaviour in North American Jerusalem crickets (
Stenopelmatus
) (Orthoptera: Stenopelmatidae).北美耶路撒冷蟋蟀的交流和生殖行为 (
Stenopelmatus
) (直翅目: Stenopelmatidae)。
PREAI
分享收藏
查看更多内容