.png){kind=icon}
返回
A lightweight and gradient-stable neural layer
.png){kind=icon}
OA
AI
DOI：10.1016/j.neunet.2024.106269.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
To enhance resource efficiency and model deployability of neural networks, we propose a neural -layer architecture based on Householder weighting and absolute -value activating, called Householder -absolute neural layer or simply Han -layer. Compared to a fully connected layer with d -neurons and d outputs, a Hanlayer reduces the number of parameters and the corresponding computational complexity from O ( d 2 ) to O ( d ) . The Han -layer structure guarantees that the Jacobian of the layer function is always orthogonal, thus ensuring gradient stability (i.e., free of gradient vanishing or exploding issues) for any Han -layer sub -networks. Extensive numerical experiments show that one can strategically use Han -layers to replace fully connected (FC) layers, reducing the number of model parameters while maintaining or even improving the generalization performance. We will also showcase the capabilities of the Han -layer architecture on a few small stylized models, and discuss its current limitations.
Keyword:
Deep neural network
Low complexity
Lightweight model
Gradient stability
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.3
论文数:
8.2K
被引数:
3.0W
机构
引用论文
The conversion of 2-cyano cyanothioformanilides into 3-aminoindole-2-carbonitriles using triphenylphosphine使用三苯基膦将2-氰基硫代甲酰苯胺转化为3-氨基吲哚-2-腈.png)
Tetrahedron
TetrahedronIF0
PREAI
分享收藏分享收藏
分享收藏分享收藏Gradient-based learning applied to document recognition基于梯度的学习在文档识别中的应用PROCEEDINGS OF THE IEEE
PROCEEDINGS OF THE IEEEIF25.9
PREAI
分享收藏Pruning and quantization for deep neural network acceleration: A survey用于深度神经网络加速的修剪和量化: 一项调查NEUROCOMPUTINGOAAI
NEUROCOMPUTINGIF6.5
OAAI分享收藏
查看更多内容