Search Results for author:
Found 21 papers, 2 papers with code
Interpolation, Approximation and Controllability of Deep Neural Networks
We investigate the expressive power of deep residual neural networks idealized as continuous dynamical systems through control theory.
On the Universal Approximation Property of Deep Fully Convolutional Neural Networks
We study the approximation of shift-invariant or equivariant functions by deep fully convolutional networks from the dynamical systems perspective.
Deep Neural Network Approximation of Invariant Functions through Dynamical Systems
We study the approximation of functions which are invariant with respect to certain permutations of the input indices using flow maps of dynamical systems.
Neural Network Architecture Beyond Width and Depth
It is proved by construction that height-$s$ ReLU NestNets with $\mathcal{O}(n)$ parameters can approximate $1$-Lipschitz continuous functions on $[0, 1]^d$ with an error $\mathcal{O}(n^{-(s+1)/d})$, while the optimal approximation error of standard ReLU networks with $\mathcal{O}(n)$ parameters is $\mathcal{O}(n^{-2/d})$.
IAE-Net: Integral Autoencoders for Discretization-Invariant Learning
Discretization invariant learning aims at learning in the infinite-dimensional function spaces with the capacity to process heterogeneous discrete representations of functions as inputs and/or outputs of a learning model.
Deep Network Approximation in Terms of Intrinsic Parameters
Furthermore, we show that the idea of learning a small number of parameters to achieve a good approximation can be numerically observed.
Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
This paper develops simple feed-forward neural networks that achieve the universal approximation property for all continuous functions with a fixed finite number of neurons.
Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
This paper concentrates on the approximation power of deep feed-forward neural networks in terms of width and depth.
Neural Network Approximation: Three Hidden Layers Are Enough
A three-hidden-layer neural network with super approximation power is introduced.
Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
More generally for an arbitrary continuous function $f$ on $[0, 1]^d$ with a modulus of continuity $\omega_f(\cdot)$, the constructive approximation rate is $\omega_f(\sqrt{d}\, N^{-\sqrt{L}})+2\omega_f(\sqrt{d}){N^{-\sqrt{L}}}$.
Optimization in Machine Learning: A Distribution Space Approach
We present the viewpoint that optimization problems encountered in machine learning can often be interpreted as minimizing a convex functional over a function space, but with a non-convex constraint set introduced by model parameterization.
Deep Network Approximation for Smooth Functions
This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously.
Deep Learning via Dynamical Systems: An Approximation Perspective
We build on the dynamical systems approach to deep learning, where deep residual networks are idealized as continuous-time dynamical systems, from the approximation perspective.
Deep Network Approximation Characterized by Number of Neurons
This paper quantitatively characterizes the approximation power of deep feed-forward neural networks (FNNs) in terms of the number of neurons.
On the Convergence and Robustness of Batch Normalization
Despite its empirical success, the theoretical underpinnings of the stability, convergence and acceleration properties of batch normalization (BN) remain elusive.
Nonlinear Approximation via Compositions
In particular, for any function $f$ on $[0, 1]$, regardless of its smoothness and even the continuity, if $f$ can be approximated using a dictionary when $L=1$ with the best $N$-term approximation rate $\varepsilon_{L, f}={\cal O}(N^{-\eta})$, we show that dictionaries with $L=2$ can improve the best $N$-term approximation rate to $\varepsilon_{L, f}={\cal O}(N^{-2\eta})$.
A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent
Despite its empirical success and recent theoretical progress, there generally lacks a quantitative analysis of the effect of batch normalization (BN) on the convergence and stability of gradient descent.
Image Restoration: A General Wavelet Frame Based Model and Its Asymptotic Analysis
In this paper, we introduce a generic wavelet frame based image restoration model, called the "general model", which includes most of the existing wavelet frame based models as special cases.
l0 Norm Based Dictionary Learning by Proximal Methods with Global Convergence
Sparse coding and dictionary learning have seen their applications in many vision tasks, which usually is formulated as a non-convex optimization problem.
Adaptive low rank and sparse decomposition of video using compressive sensing
We address the problem of reconstructing and analyzing surveillance videos using compressive sensing.
A Singular Value Thresholding Algorithm for Matrix Completion
Off-the-shelf algorithms such as interior point methods are not directly amenable to large problems of this kind with over a million unknown entries.
Optimization and Control
