Audio-guided self-supervised learning for disentangled visual speech representations

Dalu Feng

doi:10.1007/s11704-024-3787-8

Audio-guided self-supervised learning for disentangled visual speech representations

2024-06-25

0

PRE

AI

DOI：10.1007/s11704-024-3787-8

Original

Original request for help

Save

Abstract

En 中文

4 ConclusionIn this paper, we propose a novel two-branch framework to learn the disentangled visual speech representations based on two particular observations. Its main idea is to introduce the audio signal to guide the learning of speech-relevant cues and introduce a bottleneck to restrict the speech-irrelevant branch from learning high-frequency and fine-grained speech cues. Experiments on both the word-level and sentence-level audio-visual speech datasets LRW and LRS2-BBC show the effectiveness. Our future work is to explore more explicit auxiliary tasks and constraints beyond the reconstruction task of the speech-relevant and irrelevant branch to improve further its ability of capturing speech cues in the video. Meanwhile, it's also a nice try to combine multiple types of knowledge representations [10] to further boost the obtained speech epresentations, which is also left for the future work.

Audio-guided self-supervised learning for disentangled visual speech representations

Abstract

Journal

Organization