.png){kind=icon}
返回
A fine-grained deconfounding study for knowledge-based visual dialog
.png){kind=icon}
OA
AI
DOI：10.1016/j.visinf.2024.09.007.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Knowledge-based Visual Dialog is a challenging vision-language task, where an agent engages in dialog to answer questions with humans based on the input image and corresponding commonsense knowledge. The debiasing methods based on causal graphs have gradually sparked much attention in the field of Visual Dialog (VD), yielding impressive achievements. However, existing studies focus on the coarse-grained deconfounding, which lacks a principled analysis of the bias. In this paper, we propose a fined-grained study of deconfounding on: (1) We define the confounder from two perspectives. The first is user preference (denoted as Uh), derived from human-annotated dialog history, which may introduce spurious correlations between questions and answers. The second is commonsense language bias (denoted as Uc), where certain words appear so frequently in the retrieved commonsense knowledge that the model tends to memorize these patterns, thereby establishing spurious correlations between the commonsense knowledge and the answers. (2) Given that the current question directly influences answer generation, we further decompose the confounders into Uh1, Uh2 and Uc1, Uc2, based on their relevance to the current question. Specifically, Uh1 and Uc1 represent dialog history and high-frequency words that are highly correlated with the current question, while Uh2 and Uc2 are sampled from dialog history and words with low relevance to the current question. Through a comprehensive evaluation and comparison of all components, we demonstrate the necessity of jointly considering both Uh and Uc. Fine-grained deconfounding, particularly with respect to the current question, proves to be more effective. Ablation studies, quantitative results, and visualizations further confirm the effectiveness of the proposed method. (c) 2024 The Authors. Published by Elsevier B.V. on behalf of Zhejiang University and Zhejiang University (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Keyword:
Visual dialog
Knowledge
Deconfounding
Causality
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
3.9
论文数:
246
被引数:
628
机构
引用论文
分享收藏Interpretable learning based Dynamic Graph Convolutional Networks for Alzheimer's Disease analysis基于可解释学习的动态图卷积网络用于阿尔茨海默病分析.png)
INFORMATION FUSION
INFORMATION FUSIONIF15.5
PREAI
分享收藏Learning Dual Encoding Model for Adaptive Visual Understanding in Visual Dialogue视觉对话中自适应视觉理解的学习双编码模型
PREAI
分享收藏
查看更多内容