.png){kind=icon}
返回
Spatial-Temporal Graphs for Cross-Modal Text2Video Retrieval
.png){kind=icon}
PRE
AI
DOI：10.1109/TMM.2021.3090595.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Cross-modal text to video retrieval aims to find relevant videos given text queries, which is crucial for various real-world applications. The key to address this task is to build the correspondence between video and text such that related samples from different modalities can be aligned. As the text (sentence) contains both nouns and verbs representing objects as well as their interactions, retrieving relevant videos requires a fine-grained understanding of video contents-not only the semantic concepts (i.e., objects) but also the interactions between them. Nevertheless, current approaches mostly represent videos with aggregated frame-level features for the learning of joint space and ignore the information of object interactions, which usually results in suboptimal retrieval performance. To improve the performance of cross-modal video retrieval, this paper proposes a framework that models videos as spatial-temporal graphs where nodes correspond to visual objects and edges correspond to the relations/interactions between objects. With the spatial-temporal graphs, object interactions in frame sequences can be captured to enrich the video representations for joint space learning. Specifically, Graph Convolutional Network is introduced to learn the representations on spatial-temporal graphs, aiming to encode spatial-temporal interactions between objects; while BERT is introduced to dynamically encode the sentence according to the context for cross-modal retrieval. Extensive experiments verify the effectiveness of the proposed framework and it achieves promising performances on both MSR-VTT and LSMDC datasets.
Keyword:
Visualization
Semantics
Bit error rate
Encoding
Task analysis
Feature extraction
Microphones
Cross-modal retrieval
video retrieval
spatial-temporal graphs
cross-modal learning
期刊
IF:
9.7
论文数:
4.6K
被引数:
2.4W
机构
引用论文
A retrospective 5-year review of rubella in South Africa prior to the introduction of a rubella-containing vaccine.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations视觉基因组: 使用众包密集图像注释连接语言和视觉
OAAI
OAAI分享收藏
查看更多内容