.png){kind=icon}
返回
Efficient audio-visual information fusion using encoding pace synchronization for Audio-Visual Speech Separation
.png){kind=icon}
PRE
AI
DOI：10.1016/j.inffus.2024.102749.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Contemporary audio-visual speech separation (AVSS) models typically use encoders that merge audio and visual representations by concatenating them at a specific layer. This approach assumes that both modalities progress at the same pace and that information is adequately encoded at the chosen fusion layer. However, this assumption is often flawed due to inherent differences between the audio and visual modalities. In particular, the audio modality, being more directly tied to the final output (i.e., denoised speech), tends to converge faster than the visual modality. This discrepancy creates a persistent challenge in selecting the appropriate layer for fusion. To address this, we propose the Encoding Pace Synchronization Network (EPS-Net) for AVSS. EPS-Net allows for the independent encoding of the two modalities, enabling each to be processed at its own pace. At the same time, it establishes communication between the audio and visual modalities at corresponding encoding layers, progressively synchronizing their encoding speeds. This approach facilitates the gradual fusion of information while preserving the unique characteristics of each modality. The effectiveness of the proposed method has been validated through extensive experiments on the LRS2, LRS3, and VoxCeleb2 datasets, demonstrating superior performance over state-of-the-art methods.
Keyword:
Audio-visual speech separation
Encoding pace synchronization
Unimodal inter-layer synchronizer
Cross-modal intra-layer synchronizer
期刊
IF:
15.5
论文数:
4.2K
被引数:
2.7W
机构
引用论文
Solution structure of the Lewis x oligosaccharide determined by NMR spectroscopy and molecular dynamics simulations.png)
Biochemistry
BiochemistryIF0
PREAI
分享收藏Neonatal caffeine administration causes a permanent increase in the dendritic length of prefrontal cortical neurons of ratsSynapse
SynapseIF0
PREAI
分享收藏
Brain implantations of engineered gaba-releasing cells suppress tremor in an animal model of ParkinsonismNeuroscience
NeuroscienceIF0
PREAI
分享收藏
查看更多内容