.png){kind=icon}
.png){kind=icon}
Return
A lightweight and gradient-stable neural layer
.png){kind=icon}
OA
AI
DOI：10.1016/j.neunet.2024.106269.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
To enhance resource efficiency and model deployability of neural networks, we propose a neural -layer architecture based on Householder weighting and absolute -value activating, called Householder -absolute neural layer or simply Han -layer. Compared to a fully connected layer with d -neurons and d outputs, a Hanlayer reduces the number of parameters and the corresponding computational complexity from O ( d 2 ) to O ( d ) . The Han -layer structure guarantees that the Jacobian of the layer function is always orthogonal, thus ensuring gradient stability (i.e., free of gradient vanishing or exploding issues) for any Han -layer sub -networks. Extensive numerical experiments show that one can strategically use Han -layers to replace fully connected (FC) layers, reducing the number of model parameters while maintaining or even improving the generalization performance. We will also showcase the capabilities of the Han -layer architecture on a few small stylized models, and discuss its current limitations.
Keywords:
Deep neural network
Low complexity
Lightweight model
Gradient stability
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
6.3
Papers:
7.8K
Citations:
3.0W