.png){kind=icon}
Return
HyperLips: hyper control lips with high resolution decoder for talking face generation
.png){kind=icon}
PRE
AI
DOI：10.1007/s10489-024-05914-z.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Talking face generation has a wide range of potential applications in the field of virtual digital humans. However, rendering high-fidelity facial video while ensuring lip synchronization remains challenging for existing audio-driven talking face generation approaches. To tackle this problem, we present HyperLips, a two-stage framework comprising a hypernetwork for lip control and a high-resolution decoder for generating high-fidelity faces. In the first stage, we develop a base face generation network that utilizes the hypernetwork to regulate the encoding latent code of the visual face information based on audio. The process begins using FaceEncoder to extract features from the face frame, resulting in a latent code. Next, HyperConv modifies the latent code to synchronize the lip movement with the audio, with its weighting parameters updated by HyperNet using the audio features as input. Finally, the FaceDecoder decodes the modified and synchronized latent code into visual face content. In the second stage, higher-quality face videos are obtained through a high-resolution decoder. To enhance face generation quality, we trained a high-resolution decoder, HRDecoder, using face images and detected sketches generated from the first stage as input. The experimental results on benchmarks LRS2 show that the performance of HyperLips is obviously beyond the state-of-the-art methods, e.g., the PSNR, SSIM (higher is better) are improved from 33.281, 0.894 to 34.914, 0.914, respectively; LPIPS, FID, LMD (lower is better) are reduced from 0.051, 5.199, 1.471 to 0.032, 3.690, 1.186, respectively. Our method outperforms state-of-the-art with more realistic, high-fidelity, and lip-synchronized talking faces, as demonstrated by extensive quantitative and qualitative experiments. Please refer our project page as below:https://semchan.github.io/HyperLips_Project/
Keywords:
Talking face generation
Hypernetwork
Lip synchronization
High-fidelity faces
Journal
IF:
3.5
Papers:
7.6K
Citations:
1.7W
Organization
No organization information available
Cited Papers
Solution structure of the Lewis x oligosaccharide determined by NMR spectroscopy and molecular dynamics simulations.png)
Biochemistry
BiochemistryIF0
PREAI
ShareSaveShareSave
ShareSaveShareSaveOligomycin and antimycin A prevent nitric oxide–induced apoptosis by blocking cytochrome C leakage
PREAI
ShareSave
View more