.png){kind=icon}
返回
Root quantization: a self-adaptive supplement STE
.png){kind=icon}
PRE
AI
DOI：10.1007/s10489-022-03691-1.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Low precision deep neural network model quantization can further reveal stronger abilities of models such as shorter inference time, lower energy consumption and memory usage, but meanwhile induce performance degradation and instability during training. Straight Through Estimator (STE) is widely used in Quantization-Aware-Training (QAT) to overcome these shortcomings, and achieves good results on (2-, 3-, 4-bit) quantization. Different STE function may achieve different performance under various quantization precision settings. In order to explore the applicable bit-width settings range of STE functions and stabilize the training process, we propose Root Quantization. Root Quantization combines two estimators, the linear estimator and the root estimator. While linear estimator is based on existing methods of training quantizer and weights under task loss function, root estimator is based on high degree root and acts as a correction module to fine-tune the weights, which not only approximates the gradient of quantization error, but also makes the gradient more accurate. Root estimator can also adapt and adjust each layer's root degree to the most suitable value through the task loss gradient. Extensive experimental results on CIFAR-10 and ImageNet, with different network architectures under various bit-width range, show the effectiveness of our method.
Keyword:
Model compression
Model quantization
STE
Mobil computing
期刊
IF:
3.5
论文数:
7.6K
被引数:
1.7W
机构
引用论文
A Hierarchical Cardiac Rhythm Classification Methodology Based on Electrocardiogram Fiducial Points一种基于心电图基准点的分层心律分类方法
OAAI
OAAI分享收藏MXQN:Mixed quantization for reducing bit-width of weights and activations in deep convolutional neural networks.png)
APPLIED INTELLIGENCE
APPLIED INTELLIGENCEIF3.5
PREAI
分享收藏HFPQ: deep neural network compression by hardware-friendly pruning-quantizationHFPQ: 通过硬件友好的修剪量化进行深度神经网络压缩APPLIED INTELLIGENCE
APPLIED INTELLIGENCEIF3.5
PREAI
分享收藏The power of obfuscation techniques in malicious JavaScript code: A measurement study恶意JavaScript代码中混淆技术的功能: 一项测量研究OAAI
OAAI分享收藏
查看更多内容