.png){kind=icon}
返回
Generating counterfactual negative samples for image-text matching
.png){kind=icon}
PRE
AI
DOI：10.1016/j.ipm.2024.103990.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The method of image-text matching typically employs hard triplet loss as its optimization objective to learn coarse correspondences based on object co-occurrence statistics. However, due to insufficiently sampled negative instances, this coarse correspondences not only leads to the model learning biases in semantic co-occurrence but also obscures the model's understanding of crucial semantic and significant semantic contextual dependencies. In this study, we propose the Generating Feature-level and Relation-level Counterfactual Negative Samples method (GFRN) for image-text matching. This method utilizes prior knowledge and gradients to mask key regions or words to generate feature-level counterfactual negative samples, or disrupts their important contextual dependencies through Bernoulli distributions and self-supervised learning to generate relation-level counterfactual negative samples with sufficient information. Subsequently, we employ these counterfactual samples to construct contrastive triplet losses to enhance the training of the image-text matching model. Consequently, the model's ability to understand crucial semantic concepts and complex dependency relationships is significantly enhanced, and semantic biases are greatly reduced. Compared to state-of-the-art methods, the proposed GFRN improves rSum by 3.9% on Flickr30K, 2.0% on MSCOCO1K, and 4.8% on MSCOCO5K, with significant improvements in R@1 across all datasets.
Keyword:
Image-text matching
Cross-modal retrieval
Negative sample generation
Counterfactual reasoning
期刊
I
IF:
6.9
论文数:
5.2K
被引数:
1.4W
机构
引用论文
.png){kind=icon}
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations视觉基因组: 使用众包密集图像注释连接语言和视觉
OAAI
OAAI分享收藏
查看更多内容