.png){kind=icon}
返回
Cross-Utterance Conditioned VAE for Speech Generation
.png){kind=icon}
PRE
AI
DOI：10.1109/TASLP.2024.3453598.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Speech synthesis systems powered by neural networks hold promise for multimedia production, but frequently face issues with producing expressive speech and seamless editing. In response, we present the Cross-Utterance Conditioned Variational Autoencoder speech synthesis (CUC-VAE S2) framework to enhance prosody and ensure natural speech generation. This framework leverages the powerful representational capabilities of pre-trained language models and the re-expression abilities of variational autoencoders (VAEs). The core component of the CUC-VAE S2 framework is the cross-utterance CVAE, which extracts acoustic, speaker, and textual features from surrounding sentences to generate context-sensitive prosodic features, more accurately emulating human prosody generation. We further propose two practical algorithms tailored for distinct speech synthesis applications: CUC-VAE TTS for text-to-speech and CUC-VAE SE for speech editing. The CUC-VAE TTS is a direct application of the framework, designed to generate audio with contextual prosody derived from surrounding texts. On the other hand, the CUC-VAE SE algorithm leverages real mel spectrogram sampling conditioned on contextual information, producing audio that closely mirrors real sound and thereby facilitating flexible speech editing based on text such as deletion, insertion, and replacement. Experimental results on the LibriTTS datasets demonstrate that our proposed models significantly enhance speech synthesis and editing, producing more natural and expressive speech.
Keyword:
Multimedia systems
Neural networks
Natural languages
Production
Speech enhancement
Feature extraction
Acoustics
Text to speech
Mirrors
Spectrogram
Pre-trained language model
speech editing
speech synthesis
TTS
variational autoencoder
期刊
I
IF:
5.1
论文数:
2.6K
被引数:
1.1W
机构
引用论文
分享收藏Five Fabaceae Karyotype and Phylogenetic Relationship Analysis Based on Oligo-FISH for 5S rDNA and (AG3T3)3基于5S rDNA和(AG3T3)3寡核苷酸荧光原位杂交(Oligo-FISH)的五种豆科植物核型及系统发育关系分析.png)
Genes
OAAI
GenesIF0
OAAI分享收藏Biogeography of the West Indies: A complex scenario for species radiations in terrestrial and aquatic habitats西印度群岛的生物地理学：陆地和淡水栖息地物种辐射的复杂情景
OAAI
OAAI分享收藏Burden of Natural-Cause and Cause-Specific Mortality Associated with Long-Term Exposure to PM2.5: A Case Study in Attica Region, GreeceAtmosphereOAAI
AtmosphereIF0
OAAI分享收藏
查看更多内容