.png){kind=icon}
.png){kind=icon}
Return
Adversarial Contrastive Autoencoder With Shared Attention for Audio-Visual Correlation Learning
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2025.3546610.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Cross-modal audio-visual correlation learning has been an active research topic, which aims to embed audio and visual feature sequences into a common subspace where their correlation is maximized. The challenge of audio-visual correlation learning lies in two major aspects: 1) audio and visual feature sequences respectively contain different patterns belonging to different feature spaces, and 2) semantic mismatch between audio and visual sequences inevitably happens during cross-modal matching. Most existing methods only take the first aspect into account, therefore facing the difficulty in distinguishing matched and mismatched semantic correlations between audio and visual sequences. In this work, an adversarial contrastive autoencoder with a shared attention network (ACASA) is proposed for correlation learning in audio-visual retrieval. In particular, the proposed shared attention mechanism is parameterized, in which local salient information is enhanced to contribute to the final feature representation. Simultaneously, adversarial contrastive learning is exploited to maximize semantic feature consistency and improve the ability to distinguish matched and mismatched samples. Both inter-modal and intra-modal semantic information are utilized to supervise the model to learn more discriminative feature representation. Extensive experiments on the VEGAS and AVE datasets demonstrate that the proposed ACASA method outperforms state-of-the-art approaches in cross-modal audio-visual retrieval.
Keywords:
Correlation
Visualization
Semantics
Autoencoders
Attention mechanisms
Cross modal retrieval
Feature extraction
Data models
Generative adversarial networks
Unsupervised learning
Cross-modal retrieval
shared attention
adversarial learning
inter-intra modal loss
Journal
IF:
3.6
Papers:
9.8W
Citations:
29.4W
Organization
View more organizations