.png){kind=icon}
返回
Adaptive Filter Pruning via Sensitivity Feedback
.png){kind=icon}
PRE
AI
DOI：10.1109/TNNLS.2023.3246263.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Filter pruning is advocated for accelerating deep neural networks without dedicated hardware or libraries, while maintaining high prediction accuracy. Several works have cast pruning as a variant of l(1)-regularized training, which entails two challenges: 1) the l(1)-norm is not scaling-invariant (i.e., the regularization penalty depends on weight values) and 2) there is no rule for selecting the penalty coefficient to trade off high pruning ratio for low accuracy drop. To address these issues, we propose a lightweight pruning method termed adaptive sensitivity-based pruning (ASTER) which: 1) achieves scaling-invariance by refraining from modifying unpruned filter weights and 2) dynamically adjusts the pruning threshold concurrently with the training process. ASTER computes the sensitivity of the loss to the threshold on the fly (without retraining); this is carried efficiently by an application of L-BFGS solely on the batch normalization (BN) layers. It then proceeds to adapt the threshold so as to maintain a fine balance between pruning ratio and model capacity. We have conducted extensive experiments on a number of state-of-the-art CNN models on benchmark datasets to illustrate the merits of our approach in terms of both FLOPs reduction and accuracy. For example, on ILSVRC-2012 our method reduces more than 76% FLOPs for ResNet-50 with only 2.0% Top-1 accuracy degradation, while for the MobileNet v2 model it achieves 46.6% FLOPs Drop with a Top-1 Acc. Drop of only 2.77%. Even for a very lightweight classification model like MobileNet v3-small, ASTER saves 16.1% FLOPs with a negligible Top-1 accuracy drop of 0.03%.
Keyword:
Training
Adaptation models
Sensitivity
Computational modeling
Adaptive filters
Libraries
Filtering algorithms
Adaptive pruning
deep neural networks
model compression
structured model pruning
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
Facile Synthesis, Characterization, Nanocrystal Growth and Photoluminescence Properties of GeS Nanowires.png)
Nano
NanoIF0
PREAI
分享收藏Calcium-modulating cyclophilin ligand regulates membrane trafficking of postsynaptic GABAA receptors
OAAI
OAAI分享收藏ReCNAS: Resource-Constrained Neural Architecture Search Based on Differentiable Annealing and Dynamic PruningReCNAS: 基于可微退火和动态修剪的资源受限神经结构搜索
PREAI
分享收藏Mechanical, but not infective, pacemaker erosion may be successfully managed by re-implantation of pacemakers.HeartOAAI
HeartIF0
OAAI分享收藏
查看更多内容