.png){kind=icon}
.png){kind=icon}
Return
Interplay between depth and width for interpolation in neural ODEs
.png){kind=icon}
OA
AI
DOI：10.1016/j.neunet.2024.106640.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Neural ordinary differential equations have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of the role played by their architecture remains elusive. In this work, we examine the interplay between the width p and the number of transitions between layers L (corresponding to a depth of L+1). Specifically, we construct explicit controls interpolating either a finite dataset D, comprising N pairs of points in R-d, or two probability measures within a Wasserstein error margin epsilon>0. Our findings reveal a balancing trade-off between p and L, with L scaling as 1+O(N/p) for data interpolation, and as 1+O(p(-1)+(1+p)(-1)epsilon(-d)) for measures. In the high-dimensional and wide setting where d, p > N, our result can be refined to achieve L=0. This naturally raises the problem of data interpolation in the autonomous regime, characterized by L=0. We adopt two alternative approaches: either controlling in a probabilistic sense, or by relaxing the target condition. In the first case, when p = N we develop an inductive control strategy based on a separability assumption whose probability increases with d. In the second one, we establish an explicit error decay rate with respect to p which results from applying a universal approximation theorem to a custom-built Lipschitz vector field interpolating D
Keywords:
Neural ODEs
Depth
Width
Simultaneous controllability
Transport control
Wasserstein distance
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
6.3
Papers:
7.8K
Citations:
3.0W