.png){kind=icon}
返回
Simultaneously Training and Compressing Vision-and-Language Pre-Training Model
.png){kind=icon}
PRE
AI
DOI：10.1109/TMM.2022.3233258.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Model compression is an essential step for large-scale pre-training models toward practical application and deployment on the edge device. However, when conventional compression methods following 'pre-training then compressing' two-phase pipeline are applied to Vision-and-Language Pre-training (VLP) models, it will lead to a high calculation and memory overhead. In this work, we break the two-phase pipeline and propose an efficient and effective one-phase VLP model compression mechanism, named REDUCER, which stands for 'simultaneously training and compREssing' VLP model via progressive moDUle replaCing and nEtwork Rewiring. Specifically, REDUCER consists of three insightful designs. Firstly, we design a one-phase compression framework to train and compress the VLP model simultaneously to avoid the extra calculation and memory cost caused by an isolated model compression phase in the conventional two-phase pipeline. Secondly, we propose an adaptive progressive module replacing mechanism to compress the model depth free from explicit knowledge distillation losses, relieving the multi-task optimization problems. Thirdly, we integrate pruning techniques into VLP model compression to simultaneously compress the model in width and depth. Overall, we obtain a lightweight VLP model with only one pre-training phase, and it is the first one-phase compression method for VLP models. Extensive experiments have been conducted on representative VLP models, i.e., ClipBERT and VICTOR, and the experimental results show a superior trade-off between performance and efficiency.
Keyword:
Transmission line measurements
Adaptation models
Computational modeling
Training
Task analysis
Pipelines
Costs
Vision-and-Language
model compression
pre-training Model
期刊
IF:
9.7
论文数:
4.6K
被引数:
2.4W
机构
引用论文
分享收藏Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations视觉基因组: 使用众包密集图像注释连接语言和视觉
OAAI
OAAI分享收藏Mechanisms of corticosteroid action in immune thrombocytopenic purpura (ITP): experimental studies using ITP-prone mice, (NZW x BXSB) F1.png)
Blood
OAAI
BloodIF0
OAAI分享收藏
查看更多内容