.png){kind=icon}
.png){kind=icon}
Return
Audio-visual synthesis based on conditional generative adversarial networks: a multimodal digital media evaluation approach
.png){kind=icon}
PRE
AI
Z
DOI：10.7717/peerj-cs.3753.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
With the expanding application of audio-video synthesis technology across multimedia domains, generating high-quality video while ensuring precise synchronization between audio and video has emerged as a critical research challenge. Traditional audio-video synthesis approaches exhibit marked deficiencies in temporal dependency and multimodal feature integration. To address these limitations, this study introduces the Audio-Visual Enhanced Discriminator Network (AuVi-EDNet) framework, which leverages conditional Generative Adversarial Networks, Gated Recurrent Unit (GRU) networks, and multimodal fusion techniques to tackle the task of audio-video synchronization. The proposed framework enhances synchronization accuracy and naturalness by extracting audio and Mel-Frequency Cepstral Coefficient features using GRU networks, coupled with spatiotemporal features from red-green-blue image sequences. To further refine multimodal data fusion, the model incorporates an audio-video synchronization discriminator and a global spatial attention mechanism, effectively capturing the temporal dependencies inherent in both audio and video streams. Experimental results demonstrate that AuVi-EDNet surpasses traditional methods on the Lip Reading in the Wild and Lip Reading Sentences 3-TED datasets, delivering superior accuracy and robustness in audio-video synchronization and generation quality. Moreover, the analysis grounded in this framework offers vital insights for advancing audio-video synthesis technologies and research in multimodal digital-media evaluation.
Keywords:
cGAN
Audio visual fusion
MFCC
Multimodal digital media
Journal
IF:
2.5
Papers:
3.3K
Citations:
6.9K
Organization
Cited Papers