Search Results for author:
Found 64 papers, 11 papers with code
A Mean Field Analysis Of Deep ResNet And Beyond: Towards Provably Optimization Via Overparameterization From Depth
Specifically, we propose a \textbf{new continuum limit} of deep residual networks, which enjoys a good landscape in the sense that \textbf{every local minimizer is global}.
A Sinkhorn-type Algorithm for Constrained Optimal Transport
We derive the corresponding entropy regularization formulation and introduce a Sinkhorn-type algorithm for such constrained OT problems supported by theoretical guarantees.
Multidimensional unstructured sparse recovery via eigenmatrix
This note considers the multidimensional unstructured sparse recovery problems.
Convergence Analysis of Discrete Diffusion Model: Exact Implementation through Uniformization
Diffusion models have achieved huge empirical success in data generation tasks.
Ensemble-Based Annealed Importance Sampling
We discuss how the proposed algorithm can be implemented and derive a partial differential equation governing the evolution of the ensemble under the continuous time and mean-field limit.
Understanding the Generalization Benefits of Late Learning Rate Decay
Why do neural networks trained with large learning rates for a longer time often lead to better generalization?
Accelerating Sinkhorn Algorithm with Sparse Newton Iterations
To achieve possibly super-exponential convergence, we present Sinkhorn-Newton-Sparse (SNS), an extension to the Sinkhorn algorithm, by introducing early stopping for the matrix scaling steps and a second stage featuring a Newton-type subroutine.
Statistical Spatially Inhomogeneous Diffusion Inference
Inferring a diffusion equation from discretely-observed measurements is a statistical challenge of significant importance in a variety of fields, from single-molecule tracking in biophysical systems to modeling financial instruments.
Eigenmatrix for unstructured sparse recovery
This note considers the unstructured sparse recovery problems in a general form.
Multi-Objective Optimization via Wasserstein-Fisher-Rao Gradient Flow
Multi-objective optimization (MOO) aims to optimize multiple, possibly conflicting objectives with widespread applications.
When can Regression-Adjusted Control Variates Help? Rare Events, Sobolev Embedding and Minimax Optimality
We demonstrate that this kind of quadrature rule can improve the Monte Carlo rate and achieve the minimax optimal rate under a sufficient smoothness assumption.
High-dimensional density estimation with tensorizing flow
We propose the tensorizing flow method for estimating high-dimensional probability density functions from the observed data.
Synthetic Principal Component Design: Fast Covariate Balancing with Synthetic Controls
The optimal design of experiments typically involves solving an NP-hard combinatorial optimization problem.
Continuous-in-time Limit for Bayesian Bandits
The optimal policy for the limiting HJB equation can be explicitly obtained for several common bandit problems, and we give numerical methods to solve the HJB equation when an explicit solution is not available.
Why self-attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries
The knowledge consists of a set of vectors in the same embedding space as the input sequence, containing the information of the language used to process the input sequence.
Minimax Optimal Kernel Operator Learning via Multilevel Training
Learning mappings between infinite-dimensional function spaces has achieved empirical success in many disciplines of machine learning, including generative modeling, functional data analysis, causal inference, and multi-agent reinforcement learning.
Importance Tempering: Group Robustness for Overparameterized Models
In this paper, we propose importance tempering to improve the decision boundary and achieve consistently better results for overparameterized models.
Generative Modeling via Tree Tensor Network States
In this paper, we present a density estimation framework based on tree tensor-network states.
Bayesian regularization of empirical MDPs
Due to noise, the policy learnedfrom the estimated model is often far from the optimal policy of the underlying model.
Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descent
In this paper, we study the statistical limits in terms of Sobolev norms of gradient descent for solving inverse problem from randomly sampled noisy observations using a general class of objective functions.
Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes
Numerically, we observe that neural network loss functions possesses a multiscale structure, manifested in two ways: (1) in a neighborhood of minima, the loss mixes a continuum of scales and grows subquadratically, and (2) in a larger region, the loss shows several separate scales clearly.
Monte Carlo Tree Search based Hybrid Optimization of Variational Quantum Circuits
Variational quantum algorithms stand at the forefront of simulations on near-term and future fault-tolerant quantum devices.
Accelerating Primal-dual Methods for Regularized Markov Decision Processes
Entropy regularized Markov decision processes have been widely used in reinforcement learning.
Provably convergent quasistatic dynamics for mean-field two-player zero-sum games
In this paper, we study the problem of finding mixed Nash equilibrium for mean-field two-player zero-sum games.
How to Learn when Data Gradually Reacts to Your Model
A recent line of work has focused on training machine learning (ML) models in the performative setting, i. e. when the data distribution reacts to the deployed model.
Operator Shifting for Model-based Policy Evaluation
When the error is in the residual norm, we prove that the shifting factor is always positive and upper bounded by $1+O\left(1/n\right)$, where $n$ is the number of samples used in learning each row of the transition matrix.
Model-based Reinforcement Learning
reinforcement-learning
+1
A Riemannian Mean Field Formulation for Two-layer Neural Networks with Batch Normalization
Later, the infinite-width limit of the two-layer neural networks with BN is considered, and a mean-field formulation is derived for the training dynamics.
Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality
In this paper, we study the statistical limits of deep learning techniques for solving elliptic partial differential equations (PDEs) from random samples using the Deep Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs).
Approximate Newton policy gradient algorithms
This paper proposes an approximate Newton method for the policy gradient algorithm with entropy regularization.
Statistical Numerical PDE : Fast Rate, Neural Scaling Law and When it’s Optimal
In this paper, we study the statistical limits of deep learning techniques for solving elliptic partial differential equations (PDEs) from random samples using the Deep Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs).
Variational Actor-Critic Algorithms
The objective function of the variational formulation consists of two parts: one for maximizing the value function and the other for minimizing the Bellman residual.
Enterprise-Scale Search: Accelerating Inference for Sparse Extreme Multi-Label Ranking Trees
In industrial applications, these models operate at extreme scales, where every bit of performance is critical.
Combining resampling and reweighting for faithful stochastic optimization
When the loss function is a sum of multiple terms, a popular method is the stochastic gradient descent.
On Linear Stability of SGD and Input-Smoothness of Neural Networks
The multiplicative structure of parameters and input data in the first layer of neural networks is explored to build connection between the landscape of the loss function with respect to parameters and the landscape of the model function with respect to input data.
A semigroup method for high dimensional elliptic PDEs and eigenvalue problems based on neural networks
In this paper, we propose a semigroup method for solving high-dimensional elliptic partial differential equations (PDEs) and the associated eigenvalue problems based on neural networks.
How to Learn when Data Reacts to Your Model: Performative Gradient Descent
Performative distribution shift captures the setting where the choice of which ML model is deployed changes the data distribution.
Top-$k$ eXtreme Contextual Bandits with Arm Hierarchy
We show that our algorithm has a regret guarantee of $O(k\sqrt{(A-k+1)T \log (|\mathcal{F}|T)})$, where $A$ is the total number of arms and $\mathcal{F}$ is the class containing the regression function, while only requiring $\tilde{O}(A)$ computation per time step.
Computational Efficiency
Extreme Multi-Label Classification
+2
An efficient dynamical low-rank algorithm for the Boltzmann-BGK equation close to the compressible viscous flow regime
In this paper, we propose an efficient dynamical low-rank integrator that can capture the fluid limit -- the Navier-Stokes equations -- of the Boltzmann-BGK model even in the compressible regime.
Numerical Analysis Numerical Analysis Computational Physics
A Note on Optimization Formulations of Markov Decision Processes
This note summarizes the optimization formulations used in the study of Markov decision processes.
Optimization and Control
Achieving Adversarial Robustness Requires An Active Teacher
A new understanding of adversarial examples and adversarial robustness is proposed by decoupling the data generator and the label generator (which we call the teacher).
A semigroup method for high dimensional committor functions based on neural network
This paper proposes a new method based on neural networks for computing the high-dimensional committor functions that satisfy Fokker-Planck equations.
Efficient Long-Range Convolutions for Point Clouds
We showcase this framework by introducing a neural network architecture that combines LRC-layers with short-range convolutional layers to accurately learn the energy and force associated with a $N$-body potential.
Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients
We consider two commonly-used techniques, resampling and reweighting, that rebalance the proportions of the subgroups to maintain the desired objective function.
Distributed-memory $\mathcal{H}$-matrix Algebra I: Data Distribution and Matrix-vector Multiplication
We introduce a data distribution scheme for $\mathcal{H}$-matrices and a distributed-memory algorithm for $\mathcal{H}$-matrix-vector multiplication.
Numerical Analysis Distributed, Parallel, and Cluster Computing Numerical Analysis 65F99, 65Y05
Natural Gradient for Combined Loss Using Wavelets
Natural gradients have been widely used in optimization of loss functionals over probability space, with important examples such as Fisher-Rao gradient descent for Kullback-Leibler divergence, Wasserstein gradient descent for transport-related functionals, and Mahalanobis gradient descent for quadratic loss functionals.
Borrowing From the Future: Addressing Double Sampling in Model-free Control
The main idea is to borrow extra randomness from the future to approximately re-sample the next state when the underlying dynamics of the problem are sufficiently smooth.
Mirror Descent Algorithms for Minimizing Interacting Free Energy
This note considers the problem of minimizing interacting free energy.
A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
Specifically, we propose a new continuum limit of deep residual networks, which enjoys a good landscape in the sense that every local minimizer is global.
A Mean-field Analysis of Deep ResNet and Beyond:Towards Provable Optimization Via Overparameterization From Depth
Specifically, we propose a \textbf{new continuum limit} of deep residual networks, which enjoys a good landscape in the sense that \textbf{every local minimizer is global}.
Solving Inverse Wave Scattering with Deep Learning
This paper proposes a neural network approach for solving two classical problems in the two-dimensional inverse wave scattering: far field pattern problem and seismic imaging.
Solving Traveltime Tomography with Deep Learning
This paper introduces a neural network approach for solving two-dimensional traveltime tomography (TT) problems based on the eikonal equation.
Solving Optical Tomography with Deep Learning
Both the forward map from the optical properties to the albedo operator and the inverse map are high-dimensional and nonlinear.
A⋆MCTS: SEARCH WITH THEORETICAL GUARANTEE USING POLICY AND VALUE FUNCTIONS
We apply our theoretical framework to different models for the noise distribution of the policy and value network as well as the distribution of rewards, and show that for these general models, the sample complexity is polynomial in D, where D is the depth of the search tree.
Mean Field Models for Neural Networks in Teacher-student Setting
For the two-layer networks, we derive the necessary condition of the stationary distributions of the mean field equation and explain an empirical phenomenon concerning training speed differences using the Wasserstein flow description.
Meta-learning Pseudo-differential Operators with Deep Neural Networks
This paper introduces a meta-learning approach for parameterized pseudo-differential operators with deep neural networks.
Solving Electrical Impedance Tomography with Deep Learning
Both the forward map from the electrical conductivity to the DtN map and the inverse map are high-dimensional and nonlinear.
BCR-Net: a neural network based on the nonstandard wavelet form
This paper proposes a novel neural network architecture inspired by the nonstandard form proposed by Beylkin, Coifman, and Rokhlin in [Communications on Pure and Applied Mathematics, 44(2), 141-183].
A multiscale neural network based on hierarchical nested bases
In recent years, deep learning has led to impressive results in many fields.
Numerical Analysis
A multiscale neural network based on hierarchical matrices
This network generalizes the latter to the nonlinear case by introducing a local deep neural network at each spatial scale.
Numerical Analysis
Stochastic modified equations for the asynchronous stochastic gradient descent
The resulting SME of Langevin type extracts more information about the ASGD dynamics and elucidates the relationship between different types of stochastic gradient algorithms.
Solving for high dimensional committor functions using artificial neural networks
In this note we propose a method based on artificial neural network to study the transition between states governed by stochastic processes.
Solving parametric PDE problems with artificial neural networks
The representability of such quantity using a neural-network can be justified by viewing the neural-network as performing time evolution to find the solutions to the PDE.
Numerical Analysis 65Nxx
Robust and efficient multi-way spectral clustering
We present a new algorithm for spectral clustering based on a column-pivoted QR factorization that may be directly used for cluster assignment or to provide an initial guess for k-means.
Numerical Analysis Numerical Analysis Social and Information Networks Physics and Society 68W01, 65F99
A recursive skeletonization factorization based on strong admissibility
We introduce the strong recursive skeletonization factorization (RS-S), a new approximate matrix factorization based on recursive skeletonization for solving discretizations of linear integral equations associated with elliptic partial differential equations in two and three dimensions (and other matrices with similar hierarchical rank structure).
Numerical Analysis 65R20 (primary), 65F08, 65F05 (secondary)
