.png){kind=icon}
Return
Implicit Compositional Generative Network for Length-Variable Co-Speech Gesture Synthesis
.png){kind=icon}
PRE
AI
DOI：10.1109/TMM.2023.3348331.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Co-speech gesture synthesis is a practical yet challenging task that aims to generate body motion sequences in line with speech audio. Most of the existing methods can only generate the gesture sequence with a fixed number of frames, which does not satisfy the high-quality requirement of the virtual speech video in real-world applications. In this paper, we propose a novel Implicit Compositional Generative Network (ICGN) for length-variable co-speech gesture synthesis. In ICGN, the implicit neural representation is captured and optimized for a whole gesture sequence of arbitrary length with temporal embeddings. Moreover, to enforce the synthesized gestures more realistic and consistent, we compositionally generate the gesture sequence through a well-designed asymmetric two-stream network that effectively captures and utilizes the rich correlations between speech audio and human body motions. In this way, the coarse and fine-grained gestures are synthesized, respectively, according to the corresponding content-aware and emotion-aware audio components. Extensive experiments on four widely-used benchmarks demonstrate that the proposed method renders realistic human gestures and achieves the superior performance against several state-of-the-art methods.
Keywords:
Co-speech gesture synthesis
implicit neural representations
compositional generation
Journal
IF:
9.7
Papers:
4.5K
Citations:
2.4W
Organization
Cited Papers
.png){kind=icon}
ShareSaveShareSave
ShareSave
View more