.png){kind=icon}
返回
Spatial–Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation
.png){kind=icon}
OA
AI
DOI：10.1109/TIP.2023.3345652.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Video scene graph generation (VidSGG) aims to identify objects in visual scenes and infer their relationships for a given video. It requires not only a comprehensive understanding of each object scattered on the whole scene but also a deep dive into their temporal motions and interactions. Inherently, object pairs and their relationships enjoy spatial co-occurrence correlations within each image and temporal consistency/transition correlations across different images, which can serve as prior knowledge to facilitate VidSGG model learning and inference. In this work, we propose a spatial-temporal knowledge-embedded transformer (STKET) that incorporates the prior spatial-temporal knowledge into the multi-head cross-attention mechanism to learn more representative relationship representations. Specifically, we first learn spatial co-occurrence and temporal transition correlations in a statistical manner. Then, we design spatial and temporal knowledge-embedded layers that introduce the multi-head cross-attention mechanism to fully explore the interaction between visual representation and the knowledge to generate spatial- and temporal-embedded representations, respectively. Finally, we aggregate these representations for each subject-object pair to predict the final semantic labels and their relationships. Extensive experiments show that STKET outperforms current competing algorithms by a large margin, e.g., improving the mR@50 by 8.1%, 4.7%, and 2.1% on different settings over current algorithms.
Keyword:
Correlation
Visualization
Semantics
Transformers
Task analysis
Prediction algorithms
Training
Video scene graph generation
spatial-temporal knowledge learning
vision
language
期刊
IF:
13.7
论文数:
1.0W
被引数:
8.4W
机构
引用论文
Photoperiodic effects in the Djungarian hamster: one minute of light during darktime mimics influence of long photoperiods on testicular recrudescence, body weight and pelage colour.png)
Experientia
ExperientiaIF0
PREAI
分享收藏Experimental Study on the Dynamic Performance of Water‐Lubricated Rubber Bearings with Local Contact
OAAI
OAAI分享收藏Sound power of vibrating cylinders using the radiation resistance matrix and a laser vibrometer
PREAI
分享收藏Types of Parent Verbal Responsiveness That Predict Language in Young Children With Autism Spectrum Disorder预测自闭症谱系障碍幼儿语言的父母言语反应类型OAAI
OAAI分享收藏
查看更多内容