.png){kind=icon}
返回
Learning to Prompt for Vision-Language Models
.png){kind=icon}
PRE
AI
DOI：10.1007/s11263-022-01653-1.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming-one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt's context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.
期刊
IF:
9.3
论文数:
4.0K
被引数:
2.8W
机构
引用论文
分享收藏
分享收藏Synthesis of a new fluorelastomer from tetrafluoroethylene and propylene by radiation processing
PREAI
分享收藏Do Perceptions of Competence Mediate The Relationship Between Fundamental Motor Skill Proficiency and Physical Activity Levels of Children in Kindergarten?能力的感知是否可以介导幼儿园儿童的基本运动技能熟练程度与身体活动水平之间的关系？
PREAI
分享收藏The adsorption of monolayer coatings on iron nanoparticles: Mössbauer spectroscopy and XANES results
PREAI
分享收藏Role of simple descriptors and applicability domain in predicting change in protein thermostability.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏
查看更多内容