.png){kind=icon}
返回
Modeling graph-structured contexts for image captioning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.imavis.2022.104591.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The performance of image captioning has been significantly improved recently through deep neural network ar-chitectures combining with attention mechanisms and reinforcement learning optimization. Exploring visual re-lationships and interactions between different objects appearing in the image, however, is far from being investigated. In this paper, we present a novel approach that combines scene graphs with Transformer, which we call SGT, to explicitly encode available visual relationships between detected objects. Specifically, we pretrain an scene graph generation model to predict graph representations for images. After that, for each graph node, a Graph Convolutional Network (GCN) is employed to acquire relationship knowledge by aggregating the informa-tion of its local neighbors. As we train the captioning model, we feed the potential relation-aware information into the Transformer to generate descriptive sentence. Experiments on the MSCOCO dataset and the Flickr30k dataset validate the superiority of our SGT model, which can realize state-of-the-art results in terms of all the standard evaluation metrics.(c) 2022 Elsevier B.V. All rights reserved.
Keyword:
Image captioning
Transformer
Scene graph
Reinforcement learning
Attention mechanism
期刊
IF:
4.2
论文数:
4.1K
被引数:
6.7K
机构
引用论文
.png){kind=icon}
分享收藏Medical Manuscripts Impact of Hospice Enrollment on Cost and Length of Stay of a Terminal Admission
OAAI
OAAI分享收藏 没有更多内容
没有更多内容