.png){kind=icon}
Return
Explaining VQA predictions using visual grounding and a knowledge base
.png){kind=icon}
PRE
AI
DOI：10.1016/j.imavis.2020.103968.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In this work, we focus on the Visual Question Answering (VQA) task, where a model must answer a question based on an image, and the VQA-Explanations task, where an explanation is produced to support the answer. We introduce an interpretable model capable of pointing out and consuming information from a novel Knowledge Base (KB) composed of real-world relationships between objects, along with labels mined from available region descriptions and object annotations. Furthermore, this model provides a visual and textual explanations to complement the KB visualization. The use of a KB brings two important consequences: enhance predictions and improve interpretability. We achieve this by introducing a mechanism that can extract relevant information from this KB, and can point out the relations better suited for predicting the answer. A supervised attention map is generated over the KB to select the relevant relationships from it for each question-image pair. Moreover, we add image attention supervision on the explanations module to generate better visual and textual explanations. We quantitatively show that the predicted answers improve when using the KB; similarly, explanations improve with this and when adding image attention supervision. Also, we qualitatively show that the KB attention helps to improve interpretability and enhance explanations. Overall, the results support the benefits of having multiple tasks to enhance the interpretability and performance of the model. (C) 2020 Elsevier B.V. All rights reserved.
Keywords:
Deep Learning
Attention
Supervision
Knowledge Base
Interpretability
Explainability
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
4.2
Papers:
4.1K
Citations:
6.7K
Organization
Cited Papers
Awareness of activity limitations, goal setting and rehabilitation outcome in patients with brain injuries.png)
Brain Injury
Brain InjuryIF0
PREAI
ShareSaveMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
OAAI
OAAIShareSaveShareSave
ShareSave no more
no more