UnitDiff: A Unit-Diffusion Model for Code-Switching Speech Synthesis

Ke Chen

doi:10.1109/LSP.2025.3543456

UnitDiff: A Unit-Diffusion Model for Code-Switching Speech Synthesis

2025-01-01

0

PRE

AI

DOI：10.1109/LSP.2025.3543456

Original

Original request for help

Save

Abstract

En 中文

Given the scarcity of Code-Switching (CS) datasets, most researchers synthesize CS speech using multiple monolingual datasets. However, this approach presents challenges in synthesizing CS speech, such as difficulty controlling the speaker's identity and causing low intelligibility of the generated speech. This letter proposes UnitDiff, a CS speech synthesis model based on the unit-diffusion framework. The model employs the self-supervised high-level representation 'soft unit' extracted from soft HuBERT to directly predict a clean mel-spectrogram x(0). This approach enhances control over speaker identity. A language tagging method is also introduced to improve speech intelligibility. Evaluation results validate the model's effectiveness in improving the intelligibility, speaker similarity, and speaker consistency of the generated CS speech.

Keywords:

Training

Feature extraction

Predictive models

Tagging

Data models

Transformers

Switches

Speech enhancement

Noise reduction

Noise measurement

Code-switching

speech synthesis

diffusion probabilistic model

acoustic unit

UnitDiff: A Unit-Diffusion Model for Code-Switching Speech Synthesis

Abstract

Journal

Organization