Multi-Modal Multi-Channel Target Speech Separation

Rongzhi Gu

doi:10.1109/JSTSP.2020.2980956

Multi-Modal Multi-Channel Target Speech Separation

2020-03-01

62

OA

AI

DOI：10.1109/JSTSP.2020.2980956

原文链接

查看原文

摘要

En 中文

Target speech separation refers to extracting a target speaker's voice from an overlapped audio of simultaneous talkers. Previously the use of visual modality for target speech separation has demonstrated great potentials. This work proposes a general multi-modal framework for target speech separation by utilizing all the available information of the target speaker, including his/her spatial location, voice characteristics and lip movements. Also, under this framework, we investigate on the fusion methods for multi-modal joint modeling. A factorized attention-based fusion method is proposed to aggregate the high-level semantic information of multi-modalities at embedding level. This method firstly factorizes the mixture audio into a set of acoustic subspaces, then leverages the target's information from other modalities to enhance these subspace acoustic embeddings with a learnable attention scheme. To validate the robustness of proposed multi-modal separation model in practical scenarios, the system was evaluated under the condition that one of the modalities is temporarily missing, invalid or corrupted. Experiments are conducted on a large-scale audio-visual dataset collected from YouTube (to be released) that spatialized by simulated room impulse responses (RIRs). Experiment results illustrate that our proposed multi-modal framework significantly outperforms single-modal and bi-modal speech separation approaches, while can still support real-time processing.

Keyword:

Lips

Feature extraction

Acoustics

Spectrogram

Visualization

Speech processing

Robustness

Target speech separation

speech enhancement

multi-modality fusion

deep learning

Multi-Modal Multi-Channel Target Speech Separation

摘要

AI总结

期刊

机构