.png){kind=icon}
返回
RePaIR: Repaired pruning at initialization resilience
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neunet.2024.107086.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Over the past decade, the size of neural network models has gradually increased in both breadth and depth, leading to a growing interest in the application of neural network pruning. Unstructured pruning provides fine-grained sparsity and achieves better inference acceleration under specific hardware support. Unstructured Pruning at Initialization (PaI) optimizes the iterative pruning pipeline, but sparse weights increase the risk of underfitting during training. More importantly, almost all PaI algorithms focus only on obtaining the best pruning mask without considering whether the retained weights are suitable for training. Introducing Lipschitz constants during model initialization can reduce the risk of model underfitting and overfitting. As a result, we firstly analyze the impact of Lipschitz initialization on model training and propose the Repaired Initialization (ReI) algorithm for common modules with BatchNorm. Then, we utilize the same idea to repair the weight of unstructured pruned model, and name it Repaired Pruning at Initialization Resilience (RePaIR) algorithm. Extensive experiments and demonstrate that our proposed ReI and RePaIR can improve the training robustness of unpruned and pruned models, respectively, and achieve up to 1.7% accuracy gain with the same sparse pruning mask on TinyImageNet. Furthermore, we provide an improved SynFlow algorithm called Repair SynFlow (ReSynFlow), which employs Lipschitz scaling to overcome the problem of score computation in deeper models. ReSynFlow can effectively improve the maximum compression rate and is suitable for deeper models, with an accuracy improvement of up to 1.3% compared to the SynFlow algorithm on TinyImageNet.
Keyword:
Pruning at initialization
Lipschitz
Neural network
Unstructured pruning
期刊
IF:
6.3
论文数:
8.2K
被引数:
3.0W
机构
引用论文
The adsorption of divalent heavy metal ions on (8,0) carbon nanotubes: The first-principles study
PREAI
分享收藏A frequency-domain approach to improve ANNs generalization quality via proper initialization.png)
NEURAL NETWORKS
NEURAL NETWORKSIF6.3
PREAI
分享收藏分享收藏Investigation of Ti mesh-supported anodes for direct borohydride fuel cells直接硼氢化物燃料电池用钛网支撑阳极的研究
PREAI
分享收藏
On the compression of neural networks using e0-norm regularization and weight pruningNEURAL NETWORKS
OAAI
NEURAL NETWORKSIF6.3
OAAI分享收藏Developing Wind and/or Solar Powered Crop Irrigation Systems for the Great Plains为大平原开发风能和/或太阳能作物灌溉系统
PREAI
分享收藏
查看更多内容