BiC-Net: Learning Efficient Spatio-temporal Relation for Text-Video Retrieval

Ning Han

doi:10.1145/3627103

BiC-Net: Learning Efficient Spatio-temporal Relation for Text-Video Retrieval

2023-12-09

5

OA

AI

DOI：10.1145/3627103

Original request for help

Save

Abstract

En 中文

The task of text-video retrieval aims to understand the correspondence between language and vision and has gained increasing attention in recent years. Recent works have demonstrated the superiority of local spatio-temporal relation learning with graph-based models. However, most existing graph-based models are handcrafted and depend heavily on expert knowledge and empirical feedback, which may be unable to mine the high-level fine-grained visual relations effectively. These limitations result in their inability to distinguish videos with the same visual components but different relations. To solve this problem, we propose a novel cross-modal retrieval framework, Bi-Branch Complementary Network (BiC-Net), which modifies Transformer architecture to effectively bridge text-video modalities in a complementary manner via combining local spatio-temporal relation and global temporal information. Specifically, local video representations are encoded using multiple Transformer blocks and additional residual blocks to learn fine-grained spatio-temporal relations and long-term temporal dependency, calling the module a Fine-grained Spatio-temporal Transformer (FST). Global video representations are encoded using a multi-layer Transformer block to learn global temporal features. Finally, we align the spatio-temporal relation and global temporal features with the text feature on two embedding spaces for cross-modal text-video retrieval. Extensive experiments are conducted on MSR-VTT, MSVD, and YouCook2 datasets. The results demonstrate the effectiveness of our proposed model. Our code is public at https://github.com/lionel-hing/BiC-Net.

Keywords:

Text-video retrieval

spatio-temporal relation

bi-branch complementary network

BiC-Net: Learning Efficient Spatio-temporal Relation for Text-Video Retrieval

Abstract

Journal

Organization