.png){kind=icon}
返回
BiC-Net: Learning Efficient Spatio-temporal Relation for Text-Video Retrieval
.png){kind=icon}
OA
AI
DOI：10.1145/3627103.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The task of text-video retrieval aims to understand the correspondence between language and vision and has gained increasing attention in recent years. Recent works have demonstrated the superiority of local spatio-temporal relation learning with graph-based models. However, most existing graph-based models are handcrafted and depend heavily on expert knowledge and empirical feedback, which may be unable to mine the high-level fine-grained visual relations effectively. These limitations result in their inability to distinguish videos with the same visual components but different relations. To solve this problem, we propose a novel cross-modal retrieval framework, Bi-Branch Complementary Network (BiC-Net), which modifies Transformer architecture to effectively bridge text-video modalities in a complementary manner via combining local spatio-temporal relation and global temporal information. Specifically, local video representations are encoded using multiple Transformer blocks and additional residual blocks to learn fine-grained spatio-temporal relations and long-term temporal dependency, calling the module a Fine-grained Spatio-temporal Transformer (FST). Global video representations are encoded using a multi-layer Transformer block to learn global temporal features. Finally, we align the spatio-temporal relation and global temporal features with the text feature on two embedding spaces for cross-modal text-video retrieval. Extensive experiments are conducted on MSR-VTT, MSVD, and YouCook2 datasets. The results demonstrate the effectiveness of our proposed model. Our code is public at https://github.com/lionel-hing/BiC-Net.
Keyword:
Text-video retrieval
spatio-temporal relation
bi-branch complementary network
期刊
IF:
6
论文数:
2.0K
被引数:
5.4K
机构
引用论文
CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningCLIP4Clip: 端到端视频剪辑检索和字幕剪辑的实证研究.png)
NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏
分享收藏A novel and efficient xanthenic dye–organometallic ion‐pair complex for photoinitiating polymerization一种用于光引发聚合的新型高效的黄原胶染料-有机金属离子对配合物
PREAI
分享收藏
查看更多内容