.png){kind=icon}
返回
Explaining VQA predictions using visual grounding and a knowledge base
.png){kind=icon}
PRE
AI
DOI：10.1016/j.imavis.2020.103968.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In this work, we focus on the Visual Question Answering (VQA) task, where a model must answer a question based on an image, and the VQA-Explanations task, where an explanation is produced to support the answer. We introduce an interpretable model capable of pointing out and consuming information from a novel Knowledge Base (KB) composed of real-world relationships between objects, along with labels mined from available region descriptions and object annotations. Furthermore, this model provides a visual and textual explanations to complement the KB visualization. The use of a KB brings two important consequences: enhance predictions and improve interpretability. We achieve this by introducing a mechanism that can extract relevant information from this KB, and can point out the relations better suited for predicting the answer. A supervised attention map is generated over the KB to select the relevant relationships from it for each question-image pair. Moreover, we add image attention supervision on the explanations module to generate better visual and textual explanations. We quantitatively show that the predicted answers improve when using the KB; similarly, explanations improve with this and when adding image attention supervision. Also, we qualitatively show that the KB attention helps to improve interpretability and enhance explanations. Overall, the results support the benefits of having multiple tasks to enhance the interpretability and performance of the model. (C) 2020 Elsevier B.V. All rights reserved.
Keyword:
Deep Learning
Attention
Supervision
Knowledge Base
Interpretability
Explainability
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
4.2
论文数:
4.1K
被引数:
6.7K
机构
引用论文
Awareness of activity limitations, goal setting and rehabilitation outcome in patients with brain injuries.png)
Brain Injury
Brain InjuryIF0
PREAI
分享收藏Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering使VQA中的V重要: 提升图像理解在视觉问答中的作用
OAAI
OAAI分享收藏Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations视觉基因组: 使用众包密集图像注释连接语言和视觉OAAI
OAAI分享收藏
没有更多内容
没有更多内容