.png){kind=icon}
返回
Dynamic interactive learning network for audio-visual event localization
.png){kind=icon}
PRE
AI
DOI：10.1007/s10489-023-05146-7.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Audio-visual event (AVE) localization aims to detect whether an event exists in each video segment and predict its category. Only when the event is audible and visible can it be recognized as an AVE. However, sometimes the information from auditory and visual modalities is asymmetrical in a video sequence, leading to incorrect predictions. To address this challenge, we introduce a dynamic interactive learning network designed to dynamically explore the intra- and inter-modal relationships depending on the other modality for better AVE localization. Specifically, our approach involves a dynamic fusion attention of intra- and inter-modalities module, enabling the auditory and visual modalities to focus more on regions deemed informative by the other modality while focusing less on regions that the other modality considers noise. In addition, we introduce an audio-visual difference loss to reduce the distance between auditory and visual representations. Our proposed method has been demonstrated to have superior performance by extensive experimental results on the AVE dataset. The source code will be available at https://github.com/hanliang/DILN.
Keyword:
Audio-visual event localization
Dynamic fusion
Attention mechanism
Difference loss
期刊
IF:
3.5
论文数:
7.6K
被引数:
1.7W
机构
引用论文
.png){kind=icon}
Integrated compensation-based laser sensor system for in-plane and out-of-plane target tracking
PREAI
分享收藏Feature Extraction Techniques of Non-Stationary Signals for Fault Diagnosis in Machinery Systems
OAAI
OAAI分享收藏
查看更多内容