.png){kind=icon}
.png){kind=icon}
Return
Nonlinear approximation via compositions
.png){kind=icon}
OA
AI
DOI：10.1016/j.neunet.2019.07.011.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Given a function dictionary D and an approximation budget N is an element of N, nonlinear approximation seeks the linear combination of the best N terms {T-n}(1 <= n <= N) subset of D to approximate a given function f with the minimum approximation error epsilon(L,f) := min({gn}subset of R, {Tn}subset of D) parallel to f(x) - Sigma(N)(n=1)g(n)T(n)(x)parallel to. Motivated by recent success of deep learning, we propose dictionaries with functions in a form of compositions, i.e., T(x) = T-(L) circle T(L-1) circle . . . circle T-(1)(x) for all T is an element of D, and implement T using ReLU feed-forward neural networks (FNNs) with L hidden layers. We further quantify the improvement of the best N-term approximation rate in terms of N when L is increased from 1 to 2 or 3 to show the power of compositions. In the case when L > 3, our analysis shows that increasing L cannot improve the approximation rate in terms of N. In particular, for any function f on [0, 1], regardless of its smoothness and even the continuity, if f can be approximated using a dictionary when L = 1 with the best N-term approximation rate is an element of(L,f) = O(N-eta), we show that dictionaries with L = 2 can improve the best N-term approximation rate to is an element of(L,f) = O(N-2 eta). We also show that for Holder continuous functions of order alpha on [0, 1](d), the application of a dictionary with L = 3 in nonlinear approximation can achieve an essentially tight best N-term approximation rate is an element of(L,f) = O(N-2 alpha/d). Finally, we show that dictionaries consisting of wide FNNs with a few hidden layers are more attractive in terms of computational efficiency than dictionaries with narrow and very deep FNNs for approximating Holder continuous functions if the number of computer cores is larger than N in parallel computing. (C) 2019 Elsevier Ltd. All rights reserved.
Keywords:
Deep neural networks
ReLU activation function
Nonlinear approximation
Function composition
Holder continuity
Parallel computing
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
6.3
Papers:
7.8K
Citations:
3.0W
Organization
No organization information available