Search Results for author:
Found 12 papers, 0 papers with code
Deep Network Approximation: Beyond ReLU to Diverse Activation Functions
This paper explores the expressive power of deep neural networks for a diverse range of activation functions.
Why Shallow Networks Struggle with Approximating and Learning High Frequency: A Numerical Study
In this work, a comprehensive numerical study involving analysis and experiments shows why a two-layer neural network has difficulties handling high frequencies in approximation and learning when machine precision and computation cost are important factors in real practice.
On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Network
This paper explores the expressive power of deep neural networks through the framework of function compositions.
Neural Network Architecture Beyond Width and Depth
It is proved by construction that height-$s$ ReLU NestNets with $\mathcal{O}(n)$ parameters can approximate $1$-Lipschitz continuous functions on $[0, 1]^d$ with an error $\mathcal{O}(n^{-(s+1)/d})$, while the optimal approximation error of standard ReLU networks with $\mathcal{O}(n)$ parameters is $\mathcal{O}(n^{-2/d})$.
Deep Network Approximation in Terms of Intrinsic Parameters
Furthermore, we show that the idea of learning a small number of parameters to achieve a good approximation can be numerically observed.
Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
This paper develops simple feed-forward neural networks that achieve the universal approximation property for all continuous functions with a fixed finite number of neurons.
Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
This paper concentrates on the approximation power of deep feed-forward neural networks in terms of width and depth.
Neural Network Approximation: Three Hidden Layers Are Enough
A three-hidden-layer neural network with super approximation power is introduced.
Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
More generally for an arbitrary continuous function $f$ on $[0, 1]^d$ with a modulus of continuity $\omega_f(\cdot)$, the constructive approximation rate is $\omega_f(\sqrt{d}\, N^{-\sqrt{L}})+2\omega_f(\sqrt{d}){N^{-\sqrt{L}}}$.
Deep Network Approximation for Smooth Functions
This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously.
Deep Network Approximation Characterized by Number of Neurons
This paper quantitatively characterizes the approximation power of deep feed-forward neural networks (FNNs) in terms of the number of neurons.
Nonlinear Approximation via Compositions
In particular, for any function $f$ on $[0, 1]$, regardless of its smoothness and even the continuity, if $f$ can be approximated using a dictionary when $L=1$ with the best $N$-term approximation rate $\varepsilon_{L, f}={\cal O}(N^{-\eta})$, we show that dictionaries with $L=2$ can improve the best $N$-term approximation rate to $\varepsilon_{L, f}={\cal O}(N^{-2\eta})$.
