.png){kind=icon}
返回
Learning sparse reparameterization with layer-wise continuous sparsification
.png){kind=icon}
PRE
AI
DOI：10.1016/j.knosys.2023.110778.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Sparse reparameterization in Deep Neural Networks (DNNs) aims to achieve a better tradeoff between the network parameter count and performance. Recently, the lottery ticket hypothesis suppose that excellent sub-networks (winning tickets) exist in dense randomly-initialized networks. These sparse sub-networks trained from scratch are able to reach the performance of their dense counterparts. Compared with Iterative Magnitude Pruning that relies on pruning strategies, the Continuous Sparsification algorithm learns the winning tickets with gradient-based methods, achieving better performance. In this paper, we propose Layer-wise Continuous Sparsification (LCS) scheme for finding sparse sub-networks, in which the parameterized relaxation of step functions used to remove network parameters in each layer is integrated into the DNN loss as an optimization objective. LCS utilizes a family of sigmoid functions to asynchronously filter important per-layer weights throughout training, yielding sparser and better sub-networks. Experiments show that our method surpasses state-of-theart methods for sparse reparameterization. Additionally, the proposed method can be utilized as a regularization technique to further improve the accuracy of dense networks1.& COPY; 2023 Elsevier B.V. All rights reserved.
Keyword:
Deep neural network
Sparse reparameterization
Regularization technique
期刊
K
IF:
7.6
论文数:
1.2W
被引数:
4.5W
机构
引用论文
Atomic-level dispersed catalysts for PEMFCs: Progress and future prospects用于pemfc的原子级分散催化剂: 进展和未来前景.png)
EnergyChem
EnergyChemIF0
PREAI
分享收藏Developing Wind and/or Solar Powered Crop Irrigation Systems for the Great Plains为大平原开发风能和/或太阳能作物灌溉系统
PREAI
分享收藏分享收藏分享收藏
查看更多内容