.png){kind=icon}
返回
Prompt-Based Learning for Unpaired Image Captioning
.png){kind=icon}
OA
AI
DOI：10.1109/TMM.2023.3265842.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Unpaired Image Captioning (UIC) has been developed to learn image descriptions from unaligned vision-language sample pairs. Existing works usually tackle this task using adversarial learning and visual concept reward based on reinforcement learning. However, these existing works were only able to learn limited cross-domain information in vision and language domains, which restrains the captioning performance of UIC. Inspired by the success of Vision-Language Pre-Trained Models (VL-PTMs) in this research, we attempt to infer the cross-domain cue information about a given image from the large VL-PTMs for the UIC task. This research is also motivated by recent successes of prompt learning in many downstream multi-modal tasks, including image-text retrieval and vision question answering. In this work, a semantic prompt is introduced and aggregated with visual features for more accurate caption prediction under the adversarial learning framework. In addition, a metric prompt is designed to select high-quality pseudo image-caption samples obtained from the basic captioning model and refine the model in an iterative manner. Extensive experiments on the COCO and Flickr30 K datasets validate the promising captioning ability of the proposed model. We expect that the proposed prompt-based UIC model will stimulate a new line of research for the VL-PTMs based captioning.
Keyword:
Metric prompt
prompt-based learning
semantic prompt
unpaired image captioning
期刊
IF:
9.7
论文数:
4.5K
被引数:
2.4W
机构
引用论文
分享收藏Genetic Relationships Between the G Protein βγ Complex, Ste5p, Ste20p and Cdc42p: Investigation of Effector Roles in the Yeast Pheromone Response Pathway.png)
Genetics
OAAI
GeneticsIF0
OAAI分享收藏分享收藏
查看更多内容