.png){kind=icon}
.png){kind=icon}
Return
Mask-guided network for image captioning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patrec.2023.07.013.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Attention mechanisms have been widely adopted for image captioning because of their powerful performance. In this paper, we propose a Mask Captioning Network (MaC) consisting of an object layer and a background layer to capture the objects and scenes of an image to generate a sentence. To this end, we leverage the Mask RCNN to detect salient regions at the pixel level instead of a bounding box in the object layer. Meanwhile, in the background layer, a CNN model is used to encode the scene features. In addition, MaC is implemented in both LSTM-based and Transformer-based image captioning architectures. We introduce a mask-guided transformer encoder with additional features to enhance the model. Experimental results show that our model significantly outperforms (with a much richer sentence) baseline models and achieves comparable results with state-of-the-art methods on MSCOCO and Flickr30k datasets.
Keywords:
Image captioning
Deep learning
Scene understanding
Mask RCNN
Transformer