.png){kind=icon}
返回
Multi-Modal Validation and Domain Interaction Learning for Knowledge-Based Visual Question Answering
.png){kind=icon}
PRE
AI
DOI：10.1109/TKDE.2024.3384270.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Knowledge-based Visual Question Answering (KB-VQA) aims to answer the image-aware question via the external knowledge, which requires an agent to not only understand images but also explicitly retrieve and integrate knowledge facts. Intuitively, to accurately answer the question, we humans can validate the retrieved knowledge based on our memory, and then align the knowledge facts with the image regions to infer answers. However, most existing methods ignore the process of knowledge validation and alignment. In this paper, we propose the Multi-Modal Validation and Domain Interaction Learning method, which consists of two components: 1) Multi-modal validation for knowledge retrieval. We propose the multi-modal validation module (MMV) to evaluate the confidence of each retrieved knowledge fact via images and questions, which preserves knowledge candidates effective for inferring answers. 2) Domain interaction for knowledge integration. We propose the Domain Interaction TRansformer module (DI-TR) to align visual regions with knowledge facts by the interaction learning in the improved transformer. Specifically, the inter-domain and intra-domain masks are injected into each self-attention layer to control the integration scope. The proposed method outperforms several strong baselines on three widely-used knowledge-based datasets: KRVQA, OK-VQA and VQA2.0. Extensive experiments and ablation studies demonstrate the effectiveness of multi-modal knowledge validation and domain interaction learning.
Keyword:
Visualization
Knowledge based systems
Transformers
Databases
Knowledge graphs
Question answering (information retrieval)
Task analysis
Multi-modal validation
domain interaction learning
knowledge-based visual question answering
期刊
IF:
10.4
论文数:
6.8K
被引数:
3.2W
机构
引用论文
Interpretable learning based Dynamic Graph Convolutional Networks for Alzheimer's Disease analysis基于可解释学习的动态图卷积网络用于阿尔茨海默病分析.png)
INFORMATION FUSION
INFORMATION FUSIONIF15.5
PREAI
分享收藏Cubic–Quartic Optical Soliton Perturbation with Differential Group Delay for the Lakshmanan–Porsezian–Daniel Model by Lie SymmetrySymmetry
OAAI
SymmetryIF0
OAAI分享收藏分享收藏Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations视觉基因组: 使用众包密集图像注释连接语言和视觉
OAAI
OAAI分享收藏分享收藏
查看更多内容