Training Compact DNNs with l1/2 Regularization

doi:10.1016/j.patcog.2022.109206

Training Compact DNNs with l1/2 Regularization

2023-04-01

3

PRE

AI

DOI：10.1016/j.patcog.2022.109206

Original

Original request for help

Save

Abstract

En 中文

Deep neural network(DNN) has achieved unprecedented success in many fields. However, its large model parameters which bring a great burden on storage and calculation hinder the development and appli-cation of DNNs. It is worthy of compressing the model to reduce the complexity of the DNN. Sparsity -inducing regularizer is one of the most common tools for compression. In this paper, we propose utilizing the pound 1 / 2 quasi-norm to zero out weights of neural networks and compressing the networks automatically during the learning process. To our knowledge, it is the first work applying the non-Lipschitz contin-uous regularizer for the compression of DNNs. The resulting sparse optimization problem is solved by stochastic proximal gradient algorithm. For further convenience of calculation, an approximation of the threshold-form solution to the proximal operator with pound 1 / 2 is given at the same time. Extensive experi-ments with various datasets and baselines demonstrate the advantages of our new method.(c) 2022 Elsevier Ltd. All rights reserved.

Keywords:

Deep neural networks

Model compression

1

2 Quasi-norm

Non-Lipschitz regularization

Sparse optimization

Training Compact DNNs with l1/2 Regularization

Abstract

Journal

Organization