.png){kind=icon}
返回
BinaryFormer: A Hierarchical-Adaptive Binary Vision Transformer (ViT) for Efficient Computing
.png){kind=icon}
PRE
AI
DOI：10.1109/TII.2024.3396520.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Vision Transformer (ViT) has recently demonstrated impressive nonlinear modeling capabilities and achieved state-of-the-art performance in various industrial applications, such as object recognition, anomaly detection, and robot control. However, their practical deployment can be hindered by high storage requirements and computational intensity. To alleviate these challenges, we propose a binary transformer called BinaryFormer, which quantizes the learned weights of the ViT module from 32-b precision to 1 b. Furthermore, we propose a hierarchical-adaptive architecture that replaces expensive matrix operations with more affordable addition and bit operations by switching between two attention modes. As a result, BinaryFormer is able to effectively compress the model size as well as reduce the computation cost of ViT. Experimental results on the ImageNet-1K benchmark datasets show that BinaryFormer reduces the size of a typical ViT model by an average of 27.7x and converts over 99% of multiplication operations into bit operations while maintaining reasonable accuracy.
Keyword:
Transformers
Convolution
Quantization (signal)
Training
Computational modeling
Informatics
Task analysis
Binary compression
model optimization
Vision Transformer (ViT)
期刊
IF:
9.9
论文数:
8.6K
被引数:
6.0W
机构
引用论文
A Hierarchical Cardiac Rhythm Classification Methodology Based on Electrocardiogram Fiducial Points一种基于心电图基准点的分层心律分类方法
OAAI
.png){kind=icon}
OAAI分享收藏Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏Transformer-Based Imitative Reinforcement Learning for Multirobot Path Planning基于Transformer的多机器人路径规划模拟强化学习
PREAI
分享收藏
Do Perceptions of Competence Mediate The Relationship Between Fundamental Motor Skill Proficiency and Physical Activity Levels of Children in Kindergarten?能力的感知是否可以介导幼儿园儿童的基本运动技能熟练程度与身体活动水平之间的关系？
PREAI
分享收藏分享收藏
查看更多内容