Learning multimodal relationship interaction for visual relationship detection

Zhixuan Liu

doi:10.1016/j.patcog.2022.108848

2022-12-01

PRE

Zhixuan Liu

Wei‐Shi Zheng *

DOI：10.1016/j.patcog.2022.108848

原文链接

原文求助

摘要

En 中文

Visual relationship detection aims to recognize visual relationships in scenes as triplets (subject-predicate-object). Previous works have shown remarkable progress by introducing multimodal features, external linguistics, scene context, etc. Due to the loss of informative multimodal hyper-relations (i.e. relations of relationships), the meaningful contexts of relationships are not fully captured yet, which limits the reasoning ability. In this work, we propose a Multimodal Similarity Guided Relationship Interaction Network (MSGRIN) to explicitly model the relations of relationships in graph neural network paradigm. In a visual scene, the MSGRIN takes the visual relationships as nodes to construct an adaptive graph and enhances deep message passing by introducing Entity Appearance Reconstruction, Entity Relevance Filtering and Multimodal Similarity Attention. We have conducted extensive experiments on two datasets: Visual Relationship Detection (VRD) and Visual Genome (VG). The evaluation results demonstrate that the proposed MSGRIN has empirically performed more effectively overall. (C) 2022 Published by Elsevier Ltd.

Keyword:

Visual relationship detection

Scene graph generation

Relationship context

Multimodal relationship interaction

期刊

Pattern Recognition

IF:

7.6

论文数:

1.3W

被引数:

4.5W

机构

Sun Yat Sen University

学者数:

9.9W

论文数: 7.2W

被引数: 95

引用论文

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations 视觉基因组: 使用众包密集图像注释连接语言和视觉

INTERNATIONAL JOURNAL OF COMPUTER VISION

IF9.3

2017-02-06

3.1K

OAAI

Krishna, Ranjay; Zhu, Yuke; Groth, Oliver; Johnson, Justin; Hata, Kenji; Kravitz, Joshua; Chen, Stephanie; Kalantidis, Yannis; Li, Li-Jia; Shamma, David A.; Bernstein, Michael S.; Li Fei-Fei

Learning to transfer focus of graph neural network for scene graph parsing 学习转移图神经网络的焦点用于场景图解析

PATTERN RECOGNITION

IF7.6

2021-04-01

PREAI

Jiang, Junjie; He, Zaixing; Zhang, Shuyou; Zhao, Xinyue; Tan, Jianrong

A survey on still image based human action recognition

PATTERN RECOGNITION

IF7.6

2014-10-01

183

PREAI