.png){kind=icon}
返回
Nonlinear tensor train format for deep neural network compression
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neunet.2021.08.028.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep neural network (DNN) compression has become a hot topic in the research of deep learning since the scale of modern DNNs turns into too huge to implement on practical resource constrained platforms such as embedded devices. Among variant compression methods, tensor decomposition appears to be a relatively simple and efficient strategy owing to its solid mathematical foundations and regular data structure. Generally, tensorizing neural weights into higher-order tensors for better decomposition, and directly mapping efficient tensor structure to neural architecture with nonlinear activation functions, are the two most common ways. However, the considerable accuracy loss is still a fly in the ointment for the tensorizing way especially for convolutional neural networks (CNNs), while the number of studies in the mapping way is comparatively limited and corresponding compression ratio appears to be not considerable. Therefore, in this work, by researching multiple types of tensor decompositions, we realize that tensor train (TT), which has specific and efficient sequenced contractions, is potential to take into account both of tensorizing and mapping ways. Then we propose a novel nonlinear tensor train (NTT) format, which contains extra nonlinear activation functions embedded in sequenced contractions and convolutions on the top of the normal TT decomposition and the proposed TT format connected by convolutions, to compensate the accuracy loss that normal TT cannot give. Further than just shrinking the space complexity of original weight matrices and convolutional kernels, we prove that NTT can afford an efficient inference time as well. Extensive experiments and discussions demonstrate that the compressed DNNs in our NTT format can almost maintain the accuracy at least on MNIST, UCF11 and CIFAR-10 datasets, and the accuracy loss caused by normal TT could be compensated significantly on large-scale datasets such as ImageNet. (C) 2021 Published by Elsevier Ltd.
Keyword:
Tensor train decomposition
Nonlinear tensor train
Sequenced contractions
Sequenced convolutions
Neural network compression
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.3
论文数:
8.2K
被引数:
3.0W
机构
引用论文
Contributions of Valine-292 in the Nicotinamide Binding Site of Liver Alcohol Dehydrogenase and Dynamics to Catalysis,.png)
Biochemistry
BiochemistryIF0
PREAI
分享收藏分享收藏Recurrent Neural Network Model for IoT and Networking Malware Threat Detection用于物联网和网络恶意软件威胁检测的递归神经网络模型
PREAI
分享收藏Molecular Modeling of the Structural and Dynamical Changes in Calcium Channel TRPV5 Induced by the African-Specific A563T VariationBiochemistry
OAAI
BiochemistryIF0
OAAI分享收藏Heavy metal tolerance of marine phytoplankton. IV. Combined effect of zinc and cadmium on growth and uptake in some marine diatoms海洋浮游植物对重金属的耐受性。四。锌和镉对某些海洋硅藻生长和吸收的综合影响
PREAI
分享收藏DecomVQANet: Decomposing visual question answering deep network via tensor decomposition and regressionDecompvqanet: 通过张量分解和回归分解视觉问答深度网络PATTERN RECOGNITION
PATTERN RECOGNITIONIF7.6
PREAI
分享收藏
查看更多内容