.png){kind=icon}
.png){kind=icon}
Return
Effective Multimodal Encoding for Image Paragraph Captioning
.png){kind=icon}
PRE
AI
DOI：10.1109/TIP.2022.3211467.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In this paper, we present a regularization-based image paragraph generation method. We propose a novel multimodal encoding generator (MEG) to generate effective multimodal encoding that captures not only an individual sentence but also visual and paragraph-sequential information. By utilizing the encoding generated by MEG, we regularize a paragraph generation model that allows us to improve the results of the captioning model in all the evaluation metrics. With the support of the proposed MEG model for regularization, our paragraph generation model obtains state-of-the-art results on the Stanford paragraph dataset once further optimized with reinforcement learning. Moreover, we perform extensive empirical analysis on the capabilities of MEG encoding. A qualitative visualization based on t-distributed stochastic neighbor embedding (t-SNE) illustrates that sentence encoding generated by MEG captures some level of semantic information. We also demonstrate that the MEG encoding captures meaningful textual and visual information by performing multimodal sentence retrieval tasks and image instance retrieval given a paragraph query.
Keywords:
Image coding
Visualization
Encoding
Generators
Training
Image reconstruction
Decoding
Multimodal encoding generation
image paragraph captioning
text generation
autoencoder
Journal
IF:
13.7
Papers:
1.0W
Citations:
8.4W