.png){kind=icon}
返回
MAR: Masked Autoencoders for Efficient Action Recognition
.png){kind=icon}
OA
AI
DOI：10.1109/TMM.2023.3263288.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Standard approaches for video action recognition usually operate on full input videos, which is inefficient due to the widespread spatio-temporal redundancy in videos. The recent progress in masked video modelling, specifically VideoMAE, has shown the ability of vanilla Vision Transformers (ViT) to complement spatio-temporal contexts using limited visual content. Inspired by this, we propose Masked Action Recognition (MAR), which reduces redundant computation by discarding a proportion of patches and operating only on a portion of the videos. MAR includes two essential components: cell running masking and bridging classifier. Specifically, to enable the ViT to perceive the details beyond the visible patches, cell running masking is used to preserve the spatio-temporal correlations in videos. This ensures that the patches at the same spatial location can be observed in turn for easy reconstructions. Additionally, we notice that, although the partially observed features can reconstruct semantically explicit invisible patches, they fail to achieve accurate classification. To address this issue, we propose a bridging classifier that can help fill the semantic gap between the ViT encoded features used for reconstruction and the specialized features used for classification. Our proposed MAR can reduce the computational cost of ViT by 53%. Extensive experiments have demonstrated that MAR consistently outperforms existing ViT models by a notable margin. Notably, we found that a ViT-Large model fine-tuned by MAR achieves comparable performance to a ViT-Huge model fine-tuned by standard training methods on both Kinetics-400 and Something-Something v2 datasets. Moreover, the computation overhead of our ViT-Large model is only 14.5% of that of the ViT-Huge model.
Keyword:
Efficient action recognition
masked autoencoders
spatio-temporal redundancy
vision transformer
期刊
IF:
9.7
论文数:
4.5K
被引数:
2.4W
机构
引用论文
分享收藏The improved information environment as a key rationale for freedom of information reform in China
PREAI
分享收藏p8 Deficiency Causes Siderosis in Spleens and Lymphocyte Apoptosis in Acute Pancreatitis.png)
Pancreas
PancreasIF0
PREAI
分享收藏
Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏分享收藏
查看更多内容