Hypercomplex context guided interaction modeling for scene graph generation

Zheng Wang

doi:10.1016/j.patcog.2023.109634

2023-09-01

PRE

DOI：10.1016/j.patcog.2023.109634

原文链接

摘要

En 中文

Intuitively, humans can consciously and subjectively attend to the interactions between objects, and thus infer reasonable visual relations. However, mainstream approaches of Scene Graph Generation (SGG) strive to alleviate the long-tailed distribution problem with various complicated re-weighting strategies, where a simple concatenation of the refined object features is treated as the final representation of visual relations. In spite of their remarkable progress, such an operation overlooks the importance of interac-tion on relation recognition. To tackle the problem, this work devises a hyperComplex-Context guided Interaction Modeling (CCIM for short) plug-in, which can be successfully assimilated by the existing methods for performance improvement. Specifically, we first extract the contextual relation feature determined by the constraint relation ti union(head, tail) - head object - tail object. Then, we encode the features of re-lations and objects into hypercomplex space, with three imaginary components, to learn more expressive representations for SGG. Next, guided by the context, we can capture the interaction between a head or tail object and their relation through the Hamilton product. We further reinforce the interaction between enhanced hypercomplex-valued representations of the two entities with Quaternion inner product. At last, the concatenation of all components from the learned hypercomplex feature is adopted as our final relation representation. Extensive experiments on the popular benchmark Visual Genome in various exist-ing approaches demonstrate the effectiveness and generalization of our proposed model-agnostic method under comprehensive evaluation metrics.(c) 2023 Published by Elsevier Ltd.

Keyword:

Scene graph generation

Context guidence

Interaction modeling

Hypercomplex embedding

期刊

Pattern Recognition

IF:

7.6

论文数:

1.4W

被引数:

4.5W

机构

University of Queensland

学者数:

5.0W

论文数: 5.1W

被引数: 9.2W

引用论文

Predation by the swimming crab Charybdis japonica and piscivorous fishes: A major mortality factor in hatchery-reared juvenile Japanese flounder Paralichthys olivaceus released in Mano Bay, Sado Island, Japan

Fisheries Research

IF0

2008-01-01

PREAI

Hiroyuki Sudo; Naoto Kajihara; Tetsuo Fujii

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations 视觉基因组: 使用众包密集图像注释连接语言和视觉

INTERNATIONAL JOURNAL OF COMPUTER VISION

IF9.3

2017-02-06

3.1K

OAAI

Krishna, Ranjay; Zhu, Yuke; Groth, Oliver; Johnson, Justin; Hata, Kenji; Kravitz, Joshua; Chen, Stephanie; Kalantidis, Yannis; Li, Li-Jia; Shamma, David A.; Bernstein, Michael S.; Li Fei-Fei

Align R-CNN: A Pairwise Head Network for Visual Relationship Detection

IEEE TRANSACTIONS ON MULTIMEDIA

IF9.7

2022-01-01

PREAI

Tajrobehkar, Mitra; Tang, Kaihua; Zhang, Hanwang; Lim, Joo-Hwee

Configurable Graph Reasoning for Visual Relationship Detection

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

IF8.9

2022-01-01

PREAI

Zhu, Yi; Liang, Xiwen; Lin, Bingqian; Ye, Qixiang; Jiao, Jianbin; Lin, Liang; Liang, Xiaodan