.png){kind=icon}
Return
Speech-Driven Gesture Generation Using Transformer-Based Denoising Diffusion Probabilistic Models
.png){kind=icon}
OA
AI
DOI：10.1109/THMS.2024.3456085.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
While it is crucial for human-like avatars to perform co-speech gestures, existing approaches struggle to generate natural and realistic movements. In the present study, a novel transformer-based denoising diffusion model is proposed to generate co-speech gestures. Moreover, we introduce a practical sampling trick for diffusion models to maintain the continuity between the generated motion segments while improving the within-segment motion likelihood and naturalness. Our model can be used for online generation since it generates gestures for a short segment of speech, e.g., 2 s. We evaluate our model on two large-scale speech-gesture datasets with finger movements using objective measurements and a user study, showing that our model outperforms all other baselines. Our user study is based on the Metahuman platform in the Unreal Engine, a popular tool for creating human-like avatars and motions.
Keywords:
Diffusion models
Data models
Transformers
Feature extraction
Noise reduction
Avatars
Skeleton
Robots
Motion segmentation
Motion capture
Co-speech gesture
deep learning
gesture-based interaction
social interaction
Journal
IF:
4.4
Papers:
1.1K
Citations:
3.5K
Organization
View more organizations
Cited Papers
.png){kind=icon}
ShareSaveShareSave View more