.png){kind=icon}
返回
Enhancing the alignment between target words and corresponding frames for video captioning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patcog.2020.107702.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Video captioning aims at translating from a sequence of video frames into a sequence of words with the encoder-decoder framework. Hence, it is critical to align these two different sequences. Most existing methods exploit soft-attention (temporal attention) mechanism to align target words with corresponding frames, where the relevance of them merely depends on the previously generated words (i.e., language context). As we know, however, there is an inherent gap between vision and language, and most of the words in a caption belong to non-visual words (e.g. a, is, and in). Hence, merely with the guidance of the language context, existing temporal attention-based methods cannot exactly align target words with corresponding frames. In order to address this problem, we first introduce pre-detected visual tags from the video to bridge the gap between vision and language. The reason is that visual tags not only belong to textual modality, but also can convey visual information. Then, we present a Textual-Temporal Attention Model (TTA) to exactly align the target words with corresponding frames. The experimental results show that our proposed method outperforms the state-of-the-art methods on two well known datasets, i.e., MSVD and MSR-VTT.(1) (C) 2020 Elsevier Ltd. All rights reserved.
Keyword:
Video captioning
Alignment
Visual tags
Textual-temporal attention
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
7.6
论文数:
1.3W
被引数:
4.5W
机构
引用论文
Text/non-text image classification in the wild with convolutional neural networks.png)
PATTERN RECOGNITION
PATTERN RECOGNITIONIF7.6
PREAI
分享收藏Potent and stage-specific action of glutathione on the development of goat early embryos in vitro
PREAI
分享收藏分享收藏 没有更多内容
没有更多内容