.png){kind=icon}
返回
Modality-experts coordinated adaptation for large multimodal models
.png){kind=icon}
PRE
AI
DOI：10.1007/s11432-024-4234-4.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Driven by the expansion of foundation models and the increasing variety of downstream tasks, parameter-efficient fine-tuning (PEFT) methods have exhibited remarkable efficacy in the unimodal domain, effectively mitigating the consumption of computational resources. Although recent research has shifted attention to the multimodal domain and achieved efficient parametric adaptation of large multimodal models (LMMs) for downstream tasks, they still encounter two limitations: (1) low performance; (2) poor compatibility. This work proposes a modality-experts coordinated adaptation (ModeX) method for the multimodal domain, offering an effective, plug-and-play, and lightweight adaptation architecture for diverse LMMs. Specifically, ModeX adaptively coordinates different modality experts in terms of the types of network structure and input data. Besides, an effective coordinator equipped with a routing algorithm is developed for generating corresponding weights, which centers on leveraging the synergy among multimodal data. Extensive experiments on 15 multimodal downstream benchmarks and five LMMs demonstrate that ModeX is capable of seamlessly adapting to diverse LMMs, outperforms the state-of-the-art PEFT methods and even exhibits superior performance compared with full fine-tuning methods. Notably, on NLVR2 task, ModeX achieves 84.06% accuracy with only 12.0M trainable parameters, outperforming the full fine-tuning by 1.63%. Moreover, our ModeX method demonstrates superior stability and offers higher training efficiency, both in terms of training parameters and training duration. Our source code has been released at https://github.com/zhangy0822/ModeX.
Keyword:
large multimodal model
multimodal learning
vision-language pretraining
parameter-efficient fine-tuning
adapter
modality expert
期刊
IF:
7.6
论文数:
4.9K
被引数:
8.9K
机构
引用论文
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations视觉基因组: 使用众包密集图像注释连接语言和视觉
OAAI
.png){kind=icon}
OAAI分享收藏
Deep visual-linguistic fusion network considering cross-modal inconsistency for rumor detection面向谣言检测的跨模态不一致性深度视觉语言融合网络
PREAI
分享收藏 没有更多内容
没有更多内容