Search Results for author:
Found 8 papers, 0 papers with code
A Minimal Control Family of Dynamical Syetem for Universal Approximation
We prove that the control family $\mathcal{F}_1 = \mathcal{F}_0 \cup \{ \text{ReLU}(\cdot)\} $ is enough to generate flow maps that can uniformly approximate diffeomorphisms of $\mathbb{R}^d$ on any compact domain, where $\mathcal{F}_0 = \{x \mapsto Ax+b: A\in \mathbb{R}^{d\times d}, b \in \mathbb{R}^d\}$ is the set of linear maps and the dimension $d\ge2$.
Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation
In contrast, when the depth is unlimited, the width for UAP needs to be not less than the critical width $w^*_{\min}=\max(d_x, d_y)$, where $d_x$ and $d_y$ are the dimensions of the input and output, respectively.
Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions
In recent years, deep learning-based sequence modelings, such as language models, have received much attention and success, which pushes researchers to explore the possibility of transforming non-sequential problems into a sequential form.
Achieve the Minimum Width of Neural Networks for Universal Approximation
The universal approximation property (UAP) of neural networks is fundamental for deep learning, and it is well known that wide neural networks are universal approximators of continuous functions within both the $L^p$ norm and the continuous/uniform norm.
Vanilla feedforward neural networks as a discretization of dynamic systems
In this paper, we back to the classical network structure and prove that the vanilla feedforward networks could also be a numerical discretization of dynamic systems, where the width of the network is equal to the dimension of the input and output.
Optimization in Machine Learning: A Distribution Space Approach
We present the viewpoint that optimization problems encountered in machine learning can often be interpreted as minimizing a convex functional over a function space, but with a non-convex constraint set introduced by model parameterization.
On the Convergence and Robustness of Batch Normalization
Despite its empirical success, the theoretical underpinnings of the stability, convergence and acceleration properties of batch normalization (BN) remain elusive.
A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent
Despite its empirical success and recent theoretical progress, there generally lacks a quantitative analysis of the effect of batch normalization (BN) on the convergence and stability of gradient descent.
