.png){kind=icon}
Return
Re-Caption: Saliency-Enhanced Image Captioning Through Two-Phase Learning
.png){kind=icon}
PRE
AI
DOI：10.1109/TIP.2019.2928144.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Visual saliency and semantic saliency are important in image captioning. However, a single-phase image captioning model benefits little from limited saliency information without a saliency predictor. In this paper, a novel saliency-enhanced re-captioning framework via two-phase learning is proposed to enhance single-phase image captioning. In the framework, both visual and semantic saliency cues are distilled from the first-phase model and fused with the second-phase model for model self-boosting. The visual saliency mechanism can generate a saliency map and a saliency mask for an image without learning a saliency predictor. The semantic saliency mechanism sheds some lights on the properties of those words with the part-of-speech Noun in a caption. Besides, another type of saliency, sample saliency is proposed to compute the saliency degree of each sample, which is helpful for more robust image captioning. In addition, how to combine the three types of saliency for further performance boost is also examined. Our framework can treat an image captioning model as a saliency extractor, which may benefit other captioning models and the related tasks. The experimental results on both the Flickr30k and MSCOCO datasets show that the saliency-enhanced models can obtain promising performance gains.
Keywords:
Visualization
Semantics
Adaptation models
Computational modeling
Predictive models
Task analysis
Fans
Image captioning
robust estimation
saliency
salient region detection
two-phase learning
visual attribute
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
13.7
Papers:
1.0W
Citations:
8.4W
Organization
View more organizations
Cited Papers
.png){kind=icon}
ShareSaveShareSaveShareSaveShareSave View more