Self-Supervised Motion Perception for Spatiotemporal Representation Learning

doi:10.1109/TNNLS.2022.3160860

Self-Supervised Motion Perception for Spatiotemporal Representation Learning

2023-12-01

7

PRE

AI

DOI：10.1109/TNNLS.2022.3160860

Original

Original request for help

Save

Abstract

En 中文

In this study, we propose a novel pretext task and a self-supervised motion perception (SMP) method for spatiotemporal representation learning. The pretext task is defined as video playback rate perception, which utilizes temporal dilated sampling to augment video clips to multiple duplicates of different temporal resolutions. The SMP method is built upon discriminative and generative motion perception models, which capture representations related to motion dynamics and appearance from video clips of multiple temporal resolutions in a collaborative fashion. To enhance the collaboration, we further propose difference and convolution motion attention (MA), which drives the generative model focusing on motion-related appearance, and leverage multiple granularity perception (MG) to extract accurate motion dynamics. Extensive experiments demonstrate SMP's effectiveness for video motion perception and state-of-the-art performance of self-supervised representation models upon target tasks, including action recognition and video retrieval. Code for SMP is available at github.com/yuanyao366/SMP.

Keywords:

Task analysis

Switched mode power supplies

Spatiotemporal phenomena

Representation learning

Feature extraction

Dynamics

Training

Motion perception

self-supervised learning (SSL)

spatiotemporal representation learning

temporal dilated sampling

Self-Supervised Motion Perception for Spatiotemporal Representation Learning

Abstract

Journal

Organization