.png){kind=icon}
Return
Continuous frame motion sensitive self-supervised collaborative network for video representation learning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.aei.2023.101941.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Motion, as a feature of video that changes in temporal sequences, is crucial to visual understanding. The powerful video representation and extraction models are typically able to focus attention on motion features in challenging dynamic environments to complete more complex video understanding tasks. However, previous approaches discriminate mainly based on similar features in the spatial or temporal domain, ignoring the interdependence of consecutive video frames. In this paper, we propose the motion sensitive self-supervised collaborative network, a video representation learning framework that exploits a pretext task to assist feature comparison and strengthen the spatiotemporal discrimination power of the model. Specifically, we first propose the motion-aware module, which extracts consecutive motion features from the spatial regions by frame difference. The global-local contrastive module is then introduced, with context and enhanced video snippets being defined as appropriate positive samples for a broader feature similarity comparison. Finally, we introduce the snippet operation prediction module, which further assists contrastive learning to obtain more reliable global semantics by sensing changes in continuous frame features. Experimental results demonstrate that our work can effectively extract robust motion features and achieve competitive performance compared with other state-of-the-art self-supervised methods on downstream action recognition and video retrieval tasks.
Keywords:
Self-supervised representation learning
Pretext task
Global-local contrastive learning
Action recognition
Video retrieval
Journal
IF:
9.9
Papers:
4.4K
Citations:
1.7W
Organization
Cited Papers
ShareSaveDiagnostic ability of confocal near-infrared reflectance fundus imaging to detect retrograde microcystic maculopathy from chiasm compression. A comparative study with OCT findings.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAIShareSave
ShareSaveStrain-tunable electronic, elastic, and optical properties of CaI2monolayer: first-principles study
PREAI
ShareSaveShareSave
View more