Learning sparse reparameterization with layer-wise continuous sparsification

Xiaodong Wang

doi:10.1016/j.knosys.2023.110778

Return

Learning sparse reparameterization with layer-wise continuous sparsification

2023-09-01

PRE

DOI：10.1016/j.knosys.2023.110778

Original

Original request for help

Save

Abstract

En 中文

Sparse reparameterization in Deep Neural Networks (DNNs) aims to achieve a better tradeoff between the network parameter count and performance. Recently, the lottery ticket hypothesis suppose that excellent sub-networks (winning tickets) exist in dense randomly-initialized networks. These sparse sub-networks trained from scratch are able to reach the performance of their dense counterparts. Compared with Iterative Magnitude Pruning that relies on pruning strategies, the Continuous Sparsification algorithm learns the winning tickets with gradient-based methods, achieving better performance. In this paper, we propose Layer-wise Continuous Sparsification (LCS) scheme for finding sparse sub-networks, in which the parameterized relaxation of step functions used to remove network parameters in each layer is integrated into the DNN loss as an optimization objective. LCS utilizes a family of sigmoid functions to asynchronously filter important per-layer weights throughout training, yielding sparser and better sub-networks. Experiments show that our method surpasses state-of-theart methods for sparse reparameterization. Additionally, the proposed method can be utilized as a regularization technique to further improve the accuracy of dense networks1.& COPY; 2023 Elsevier B.V. All rights reserved.

Keywords:

Deep neural network

Sparse reparameterization

Regularization technique