Conditional Temporal Variational AutoEncoder for Action Video Prediction

doi:10.1007/s11263-023-01832-8

Conditional Temporal Variational AutoEncoder for Action Video Prediction

2023-06-18

1

PRE

AI

DOI：10.1007/s11263-023-01832-8

Original

Original request for help

Save

Abstract

En 中文

To synthesize a realistic action sequence based on a single human image, it is crucial to model both motion patterns and diversity in the action video. This paper proposes an Action Conditional Temporal Variational AutoEncoder (ACT-VAE) to improve motion prediction accuracy and capture movement diversity. ACT-VAE predicts pose sequences for an action clip from a single input image. It is implemented as a deep generative model that maintains temporal coherence according to the action category with a novel temporal modeling on latent space. Further, ACT-VAE is a general action sequence prediction framework. When connected with a plug-and-play Pose-to-Image network, ACT-VAE can synthesize image sequences. Extensive experiments bear out our approach can predict accurate pose and synthesize realistic image sequences, surpassing state-of-the-art approaches. Compared to existing methods, ACT-VAE improves model accuracy and preserves diversity.

Keywords:

Variational AutoEncoder

Action modeling

Temporal coherence

Adversarial learning

Conditional Temporal Variational AutoEncoder for Action Video Prediction

Abstract

Journal

Organization