.png){kind=icon}
返回
Parameter-Efficient Transfer Learning for Medical Visual Question Answering
.png){kind=icon}
PRE
AI
DOI：10.1109/TETCI.2023.3311333.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The Contrastive Language-Image Pre-Training (CLIP) model, pretrained on large visual text corpora, has demonstrated significant improvements in visual and linguistic tasks and has been applied to various downstream tasks. At least two issues, however, hinder the transfer learning with such powerful pretrained models in the field of medical visual question answering (Med-VQA). That current methods tend to full fine-tune these large-scale models are suffering from increasingly expensive computational cost as the model size grows in development. Additionally, published Med-VQA datasets are small that may lead to overfitting when directly fine-tuning on them. In this article, we integrate two designs and propose an efficient transfer learning method for Med-VQA named VQA-Adapter. To alleviate training costs, we introduce a novel and parameter-efficient adapter component into Med-VQA. During training, only the proposed light-weight adapter needs to be tuned, while all parameters in the large-scale visual model of CLIP could be kept frozen. We further design a multi-stage label smoothing paradigm for Med-VQA to deal with the overfitting issue in small Med-VQA datasets. Experimental results on two popular Med-VQA datasets, i.e., VQA-RAD and SLAKE, demonstrate that our method can significantly outperform existing state-of-the-art methods on both open-ended and closed-ended question answering tasks. Furthermore, compared to directly fine-tuning the entire CLIP model, our approach only requires to update 2.38% of the parameters. Extensive ablation studies, analysis and visualizations convincingly demonstrate the great potential of designing light-weight frameworks to transfer large-scale pretrained models from natural vision-language tasks to domain-specific medical applications.
Keyword:
Visualization
Task analysis
Adaptation models
Training
Computational modeling
Feature extraction
Transfer learning
Parameter-Efficient transfer learning
Adapter
CLIP
multi-stage label smoothing
medical visual question answering
期刊
I
IF:
6.5
论文数:
1.4K
被引数:
4.5K
机构
引用论文
分享收藏
TIME SERIES OF IMAGES TO IMPROVE TREE SPECIES CLASSIFICATION.png)
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences
OAAI
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information SciencesIF0
OAAI分享收藏CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningCLIP4Clip: 端到端视频剪辑检索和字幕剪辑的实证研究NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏分享收藏
查看更多内容