Blind Audiovisual Source Separation Based on Sparse Redundant Representations

doi:10.1109/TMM.2010.2050650

Blind Audiovisual Source Separation Based on Sparse Redundant Representations

2010-08-01

57

OA

AI

G

Gianluca Monaci

V

Vandergheynst, Pierre

R

Rémi Gribonval

DOI：10.1109/TMM.2010.2050650

Original

Original request for help

Save

Abstract

En 中文

In this paper, we propose a novel method which is able to detect and separate audiovisual sources present in a scene. Our method exploits the correlation between the video signal captured with a camera and a synchronously recorded one-microphone audio track. In a first stage, audio and video modalities are decomposed into relevant basic structures using redundant representations. Next, synchrony between relevant events in audio and video modalities is quantified. Based on this co-occurrence measure, audiovisual sources are counted and located in the image using a robust clustering algorithm that groups video structures exhibiting strong correlations with the audio. Next periods where each source is active alone are determined and used to build spectral Gaussian mixture models (GMMs) characterizing the sources acoustic behavior. Finally, these models are used to separate the audio signal in periods during which several sources are mixed. The proposed approach has been extensively tested on synthetic and natural sequences composed of speakers and music instruments. Results show that the proposed method is able to successfully detect, localize, separate, and reconstruct present audiovisual sources.

Keywords:

Audiovisual processing

blind source separation

Gaussian mixture models

sparse signal representation

Blind Audiovisual Source Separation Based on Sparse Redundant Representations

Abstract

Journal

Organization