.png){kind=icon}
返回
Sparse low rank factorization for deep neural network compression
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2020.02.035.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Storing and processing millions of parameters in deep neural networks is highly challenging during the deployment of model in real-time application on resource constrained devices. Popular low-rank approximation approach singular value decomposition (SVD) is generally applied to the weights of fully connected layers where compact storage is achieved by keeping only the most prominent components of the decomposed matrices. Years of research on pruning-based neural network model compression revealed that the relative importance or contribution of each neuron in a layer highly vary among each other. Recently, synapses pruning has also demonstrated that having sparse matrices in network architecture achieve lower space and faster computation during inference time. We extend these arguments by proposing that the low-rank decomposition of weight matrices should also consider significance of both input as well as output neurons of a layer. Combining the ideas of sparsity and existence of unequal contributions of neurons towards achieving the target, we propose sparse low rank (SLR) method which sparsifies SVD matrices to achieve better compression rate by keeping lower rank for unimportant neurons. We demonstrate the effectiveness of our method in compressing famous convolutional neural networks based image recognition frameworks which are trained on popular datasets. Experimental results show that the proposed approach SLR outperforms vanilla truncated SVD and a pruning baseline, achieving better compression rates with minimal or no loss in the accuracy. Code of the proposed approach is avaialble at https://github.com/sridarah/slr. (C) 2020 Elsevier B.V. All rights reserved.
Keyword:
Low-rank approximation
Singular value decomposition
Sparse matrix
Deep neural networks
Convolutional neural networks
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
Graph Regularized Non-Negative Low-Rank Matrix Factorization for Image Clustering用于图像聚类的图正则化非负低秩矩阵分解
PREAI
分享收藏Structured Pruning of Convolutional Neural Networks via L1 Regularization基于L1正则化的卷积神经网络结构修剪.png)
IEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAI分享收藏Gradient-based learning applied to document recognition基于梯度的学习在文档识别中的应用PROCEEDINGS OF THE IEEE
PROCEEDINGS OF THE IEEEIF25.9
PREAI
分享收藏分享收藏分享收藏
查看更多内容