.png){kind=icon}
返回
AudioVisual Video Summarization
.png){kind=icon}
OA
AI
DOI：10.1109/TNNLS.2021.3119969.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Audio and vision are two main modalities in video data. Multimodal learning, especially for audiovisual learning, has drawn considerable attention recently, which can boost the performance of various computer vision tasks. However, in video summarization, most existing approaches just exploit the visual information while neglecting the audio information. In this brief, we argue that the audio modality can assist vision modality to better understand the video content and structure and further benefit the summarization process. Motivated by this, we propose to jointly exploit the audio and visual information for the video summarization task and develop an audiovisual recurrent network (AVRN) to achieve this. Specifically, the proposed AVRN can be separated into three parts: 1) the two-stream long-short term memory (LSTM) is used to encode the audio and visual feature sequentially by capturing their temporal dependency; 2) the audiovisual fusion LSTM is used to fuse the two modalities by exploring the latent consistency between them; and 3) the self-attention video encoder is adopted to capture the global dependency in the video. Finally, the fused audiovisual information and the integrated temporal and global dependencies are jointly used to predict the video summary. Practically, the experimental results on the two benchmarks, i.e., SumMe and TVsum, have demonstrated the effectiveness of each part and the superiority of AVRN compared with those approaches just exploiting visual information for video summarization.
Keyword:
Visualization
Task analysis
Feature extraction
Streaming media
Semantics
Manuals
Generators
Audiovisual learning
multimodal learning
recurrent network
video summarization
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
4-Phenylbutyrate Attenuates the ER Stress Response and Cyclic AMP Accumulation in DYT1 Dystonia Cell Models.png)
PLoS ONEOAAI
PLoS ONEIF0
OAAI分享收藏Spin transport in antiferromagnetic NiO and magnetoresistance in Y3Fe5O12/NiO/Pt structuresAIP Advances
OAAI
AIP AdvancesIF0
OAAI分享收藏
查看更多内容