.png){kind=icon}
.png){kind=icon}
Return
Compositional Function Spaces for Deep Learning
.png){kind=icon}
PRE
AI
R
R
DOI：10.1137/25M1802948.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
We present a variational framework for studying functions learned by deep neural networks with rectified linear unit nonlinearities. We introduce a function space built from compositions of functions of second-order Radon-domain bounded variation. The compositional form of these functions captures the structure of deep neural networks. We prove a representer theorem that shows that deep neural networks with finite width solve regularized data-fitting problems over this space. The critical width is controlled by the square of the number of training data. This perspective explains the effect of weight-decay regularization in neural network training, the importance of skip connections, and the role of sparsity in neural networks. By considering the function-space perspective, we provide sharp links between deep learning and variational methods.
Keywords:
deep learning
neural networks
regularization
representer theorem
sparsity
Journal
IF:
6.1
Papers:
888
Citations:
1.2W
Organization
Cited Papers