Multimodal large model pretraining, adaptation and efficiency optimization

Lixia Ji

doi:10.1016/j.neucom.2024.129138

2025-02-01

PRE

DOI：10.1016/j.neucom.2024.129138

原文链接

原文求助

摘要

En 中文

Multimodal large models, leveraging extensive datasets and parameters, have provided superior solutions for multimodal tasks and have been widely applied across various domains in everyday life. However, the development of multimodal large models also faces numerous challenges. Among the most critical challenges are how to scientifically and rationally pre-train multimodal large models and how to apply these pre- trained models to downstream tasks more effectively and at a lower cost. In response to these challenges, this paper initially analyzes and summarizes various multimodal pre-training methods employed by stateof-the-art multimodal large models. By analyzing the characteristics these methods, the paper elucidates the different capabilities imparted to multimodal large models by various multimodal pre-training methods. This provides a basis for researchers to select multimodal pre-training methods according to specific needs. Subsequently, the paper categorizes and outlines general adaptation methods for downstream tasks using multimodal pre-trained models. By comparing their advantages and disadvantages, the distinct characteristic of each method are highlighted. Furthermore, we categorize multimodal downstream tasks into five types. Then, for different types of tasks based on their characteristics and application scenarios, we outline suitable adaptation methods, offering researchers amore definitive direction for adaptation studies related to various types of tasks. Additionally, from the perspectives of parameter efficiency, memory efficiency, and efficiency of data utilization, a comparative analysis of research on the efficiency optimization of multimodal large models is conducted. In conclusion, the paper summarizes the shortcomings of existing research and looks forward to the issues and development directions that need to be addressed for multimodal large models. It points out that integrating more modalities into multimodal large models, reducing resource consumption costs, enhancing model explainability and transparency, and addressing privacy and security issues will be hotspots for future research.

Keyword:

Multi-modal learning

Pretrained model

Adaptation fine-tuning

Efficient training