.png){kind=icon}
返回
Relation-aware attention for video captioning via graph learning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patcog.2022.109204.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Video captioning often uses an attentive encoder-decoder as the baseline model. However, the conven-tional attention mechanism still remains two problems. First, the attended visual feature is often irrele-vant to the target word state, because the attention process only uses the unidirectional flow from vision to linguistics, while lacking the reverse flow. Second, each attention result is independent, because it is computed only based on the previous word states while not considering the attention information from the past and future. This does not suit the attention habits of human beings. In this paper, we improve the conventional attention mechanism to a relation-aware attention mechanism. To this end, we propose two kinds of graph learning strategies, namely the linguistics-to-vision heterogeneous graph (HTG) and the vision-to-vision homogeneous graph (HMG). The HTG aims to enhance the inter-relation of attention by reversely modeling the relation of each word with respect to every attended visual feature, support-ing proper semantic alignment in between. The HMG aims to enhance the intra-relation of attention by capturing the relations among all of the attended visual features, which can leverage the attention infor-mation from the past and future to guide the current attention process. Extensive experiments on two public datasets show that our proposed method not only significantly improves the baseline model, but also outperforms state-of-the-art methods.(c) 2022 Elsevier Ltd. All rights reserved.
Keyword:
Video captioning
Relation -aware attention
Graph learning
期刊
IF:
7.6
论文数:
1.3W
被引数:
4.5W
机构
引用论文
分享收藏
Enhancing the alignment between target words and corresponding frames for video captioning.png)
PATTERN RECOGNITION
PATTERN RECOGNITIONIF7.6
PREAI
分享收藏
Potent and stage-specific action of glutathione on the development of goat early embryos in vitro
PREAI
分享收藏
查看更多内容