Align R-CNN: A Pairwise Head Network for Visual Relationship Detection

doi:10.1109/TMM.2021.3062543

Align R-CNN: A Pairwise Head Network for Visual Relationship Detection

2022-01-01

5

PRE

AI

DOI：10.1109/TMM.2021.3062543

Original

Original request for help

Save

Abstract

En 中文

Scene graphs connect individual objects with visual relationships. They serve as a comprehensive scene representation for downstream multimodal tasks. However, by exploring recent progress in Scene Graph Generation (SGG), we find that the performance of recent works is highly limited by the pairwise relationship modeling by naive feature concatenation. Such pairwise features lack sufficient object interaction due to the mis-aligned object parts, resulting in non-discriminative pairwise features for visual relationship prediction. For example, naive concatenated pairwise feature usually make the model fail to discriminate between riding and feeding for object pair person and horse. To this end, we design a meta-architecture- learning-to-align - for dynamic object feature concatenation. We call our model: Align R-CNN. Specifically, we introduce a novel attention-based multiple region alignment module that can be jointly optimized with SGG. Experiments on the large-scale SGG benchmark Visual Genome show that the proposed Align R-CNN can replace the naive feature concatenation and thus boost all the existing SGG methods.

Keywords:

Feature extraction

Visualization

Task analysis

Predictive models

Semantics

Context modeling

Data mining

Pairwise feature alignment

scene graph generation

visual attention

Align R-CNN: A Pairwise Head Network for Visual Relationship Detection

Abstract

Journal

Organization