.png){kind=icon}
返回
USEV: Universal Speaker Extraction With Visual Cue
.png){kind=icon}
OA
AI
DOI：10.1109/TASLP.2022.3205759.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
A speaker extraction algorithm seeks to extract the target speaker's speech from a multi-talker speech mixture. The prior studies focus mostly on speaker extraction from a highly overlapped multi-talker speech mixture. However, the target-interference speaker overlapping ratios could vary over a wide range from 0% to 100% in natural speech communication, furthermore, the target speaker could be absent in the speech mixture, the speech mixtures in such universal multi-talker scenarios are described as general speech mixtures. The speaker extraction algorithm requires an auxiliary reference, such as a video recording or a pre-recorded speech, to form top-down auditory attention on the target speaker. We advocate that a visual cue, i.e., lip movement, is more informative than an audio cue, i.e., pre-recorded speech, to serve as the auxiliary reference for speaker extraction in disentangling the target speaker from a general speech mixture. In this paper, we propose a universal speaker extraction network with a visual cue, that works for all multi-talker scenarios. In addition, we propose a scenario-aware differentiated loss function for network training, to balance the network performance over different target-interference speaker pairing scenarios. The experimental results show that our proposed method outperforms various competitive baselines for general speech mixtures in terms of signal fidelity.
Keyword:
General speech mixture
multi-modal
scenarioaware differentiated loss
sparsely overlapped speech
target speaker extraction
期刊
I
IF:
5.1
论文数:
2.6K
被引数:
1.1W
机构
引用论文
.png){kind=icon}
分享收藏分享收藏A Network for Advancing Dendrochronology, Dendrochemistry and Dendrohydrology in South America一个推动南美洲树轮年代学、树轮化学和树轮水文学发展的网络
PREAI
分享收藏Design of Pulse Waveform for Waveform Division Multiple Access UWB Wireless Communication System脉冲波形设计用于波形分割多址接入超宽带无线通信系统
OAAI
OAAI分享收藏Use of an Analog Task to Study Effects of Diazepam on Taste Perception, Consummatory Behaviors, and Risk Taking in a Social Context使用模拟任务研究地西泮对味觉感知、摄食行为及社交情境中风险承担的影响
PREAI
分享收藏The shadow economy in three Mediterranean countries: France, Spain and Greece. A MIMIC approach
PREAI
分享收藏
查看更多内容