.png){kind=icon}
Return
Learning Sparse Neural Networks Using Non-Convex Regularization
.png){kind=icon}
PRE
AI
DOI：10.1109/TETCI.2021.3058672.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Deep Neural Networks (DNNs) is the computing paradigm that has achieved remarkable success in various fields of engineering in recent years, primarily visual recognition. DNNs owe its success to the presence of large number of weight parameters (and increased depth), which led to huge computation and memory costs for implementation. These limiting factors hinder the scalability of such algorithms on resource constraint devices (like IoT devices). In general, DNNs are believed to be over parametrized i.e., the parameters are highly redundant, thus can be structurally removed without significant loss of performance. To solve these issues, we propose to use non-convex T$\ell _{1}$ regularizer along with the additional effect of sparse group lasso to completely remove the redundant neurons/filters that is, to introduce structured sparsity. The network has been trained using the proximal gradient method, which is useful in optimizing functions with the combination of smooth and non-smooth terms. We show that proposed regularizer manages to achieve competitive performances as well as extremely compact networks. Detailed experiments are performed on several benchmark datasets that illustrate the efficiency of the approach. On the ImageNet dataset, our approach removes more than 50% of parameters of convolutional layers and 85% parameters of fully connected layers of Alexnet with no drop in accuracy.
Keywords:
Biological neural networks
Neurons
Memory management
Internet of Things
Feature extraction
Jamming
Benchmark testing
Deep neural networks
regularization
sparse group lasso
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
I
IF:
6.5
Papers:
1.4K
Citations:
4.5K
Organization
Cited Papers
ShareSaveShareSaveShareSaveShareSaveShareSaveShareSaveA Survey on Nonconvex Regularization-Based Sparse and Low-Rank Recovery in Signal Processing, Statistics, and Machine Learning.png)
IEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAIShareSaveShareSaveShareSave
View more