.png){kind=icon}
Return
Multi-Relation Learning Network for audio-visual event localization
.png){kind=icon}
PRE
AI
DOI：10.1016/j.knosys.2024.112925.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In audio-visual event localization task, mining the rich relations within the video is essential for accurate event localization. However, the visual relations between visual entities (i.e. objects and regions) are ignored by existing methods. In addition, existing models usually create two modality-specific streams to process the intraand cross-modal relations for audio and visual features. Unfortunately, such a dual-stream architecture may exist parameter redundancy. Furthermore, the inherent temporal continuity relation between successive event segments is not explored by existing methods. To address these issues and improve event localization accuracy, we propose a Multi-Relation Learning Network (MRLN) that effectively mines visual relations, intraand cross-modal relations, and temporal continuity relations. Specifically, we propose an audio-conditioned graph attention to dynamically capture the audio-related visual relations for the visual-level relation learning. To reduce the parameter redundancy when learning the intraand cross-modal relations, a coupled modalities and relations encoder (CMRE) with a lightweight single-stream structure is introduced, where the audio-visual modalities and the intraand cross-modal relations are synchronously processed by performing parameter- shared attention. For arbitrary adjacent event segments, we propose a novel continuity loss to simulate their continuity relations, which improves the consistency of successive event segments. Moreover, MRLN is a robust framework that can be applied to dual-grained visual entities: the fine-grained object overlooked by existing works and the coarse-grained regions. Extensive experiments on benchmark datasets demonstrate the effectiveness of our method. Compared to recent state-of-the-art (SOTA) models, our method achieves the new SOTA performance while possessing significantly fewer trainable parameters.
Keywords:
Audio-visual event localization
Attention mechanism
Intra- and cross-modal relations
Temporal continuity relation
Visual relation
Journal
K
IF:
7.6
Papers:
1.3W
Citations:
4.5W
Organization
No organization information available
Cited Papers
.png){kind=icon}
ShareSave
ShareSaveShareSaveShareSave
ShareSave
View more