.png){kind=icon}
返回
Contrastive topic-enhanced network for video captioning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2023.121601.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In the field of video captioning, recent works usually focus on multi-modal video content understanding, in which transcripts are extracted from speech and are often adopted as an informational supplement. However, most existing works only consider transcripts as a supplementary modality, neglecting their potential in capturing high-level semantics, such as multi-modal topics. In fact, transcripts, as a textual attribute derived from the video, reflect the same high-level topics as the video content. Nonetheless, how to resolve the heterogeneity of multi-modal topics is still under-investigated and worth exploring. In this paper, we introduce a contrastive topic-enhanced network to consistently model heterogeneous topics, that is, inject an alignment module in advance, to learn a comprehensive latent topic space and guide caption generation. Specifically, our method includes a local semantic alignment module and a global topic fusion module. In the local semantic alignment module, a fine-grained semantic alignment at the clip-sentence granularity reduces the semantic gap between modalities. Extensive experiments have verified the effectiveness of our solution.
Keyword:
Video captioning
Multi-modal topic
Contrastive learning
Multi-modal video understanding
期刊
IF:
7.5
论文数:
3.0W
被引数:
10.2W
机构
引用论文
Interval-enhanced Graph Transformer solution for session-based recommendation用于基于会话的推荐的区间增强型图转换器解决方案
PREAI
分享收藏Social Neighborhood Graph and Multigraph Fusion Ranking for Multifeature Image Retrieval用于多特征图像检索的社会邻域图和多图融合排序
PREAI
分享收藏Multimodal graph neural network for video procedural captioning用于视频过程字幕的多模态图神经网络.png)
NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏
查看更多内容