.png){kind=icon}
返回
Efficient Video Transformers via Spatial-temporal Token Merging for Action Recognition
.png){kind=icon}
PRE
AI
DOI：10.1145/3633781.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Transformer has exhibited promising performance in various video recognition tasks but brings a huge computational cost in modeling spatial-temporal cues. This work aims to boost the efficiency of existing video transformers for action recognition through eliminating redundancies in their tokens and efficiently learning motion cues of moving objects. We propose a lightweight and plug-and-play module, namely Spatial-temporal Token Merger (STTM), to merge the tokens belonging to the same object into a more compact representation. STTM first adaptively identifies crucial object clues underlying the video as meta tokens. Similarity scores between input tokens and meta tokens are hence computed and used to guide the fusion of similar tokens in both spatial and temporal domains, respectively. To compensate for motion cues lost in the merging procedure, we compute the linear aggregation of spatial-temporal positions of tokens as motion features. STTM hence outputs a compact set of tokens fusing both appearance and motion features of moving objects. This procedure substantially decreases the number of tokens that need to be processed by each Transformer block and boosts the efficiency. As a general module, STTM can be applied to different layers of various video Transformers. Extensive experiments on the action recognition datasets Kinectics-400 and SSv2 demonstrate its promising performance. For example, it reduces the computation complexity of ViT by 38% while maintaining a similar performance on Kinectics-400. It also brings 1.7% gains of top-1 accuracy on SSv2 under the same computational cost.
Keyword:
Efficient video recognition
deep learning
transformer
spatial-temporal information
期刊
IF:
6
论文数:
2.0K
被引数:
5.4K
机构
引用论文
Exploiting Attention-Consistency Loss For Spatial-Temporal Stream Action Recognition利用注意力一致性损失进行时空流动作识别
OAAI
OAAI分享收藏Using SEPIC Topology for Improving Power Factor in Distributed Power Supply Systems.png)
EPE Journal
EPE JournalIF0
PREAI
分享收藏
查看更多内容