Search Results for author:
Found 43 papers, 10 papers with code
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates.
Rethinking Specificity in SBDD: Leveraging Delta Score and Energy-Guided Diffusion
In the field of Structure-based Drug Design (SBDD), deep learning-based generative models have achieved outstanding performance in terms of docking score.
The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling
Surprisingly, the improvement persists when we increase the number of sampling steps and can even surpass the best result from EDM-2 (1. 58) with only 39 NFEs (1. 57).
Better Neural PDE Solvers Through Data-Free Mesh Movers
Based on DMM, to efficiently and accurately model dynamic systems, we develop a moving mesh based neural PDE solver (MM-PDE) that embeds the moving mesh with a two-branch architecture and a learnable interpolation framework to preserve information within the data.
Deciphering and integrating invariants for neural operator learning with various physical mechanisms
To this end, we propose Physical Invariant Attention Neural Operator (PIANO) to decipher and integrate the physical invariants (PI) for operator learning from the PDE series with various physical mechanisms.
Sliced Denoising: A Physics-Informed Molecular Pre-Training Method
By aligning with physical principles, SliDe shows a 42\% improvement in the accuracy of estimated force fields compared to current state-of-the-art denoising methods, and thus outperforms traditional baselines on various molecular property prediction tasks.
SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models
Based on our analysis, we propose SA-Solver, which is an improved efficient stochastic Adams method for solving diffusion SDE to generate data with high quality.
Ranked #11 on
Image Generation
on ImageNet 512x512
Fractional Denoising for 3D Molecular Pre-training
Theoretically, the objective is equivalent to learning the force field, which is revealed helpful for downstream tasks.
Power-law Dynamic arising from machine learning
We study a kind of new SDE that was arisen from the research on optimization in machine learning, we call it power-law dynamic because its stationary distribution cannot have sub-Gaussian tail and obeys power-law.
Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions
We provide a simple convergence proof for AdaGrad optimizing non-convex objectives under only affine noise variance and bounded smoothness assumptions.
A new perspective on building efficient and expressive 3D equivariant graph neural networks
Geometric deep learning enables the encoding of physical symmetries in modeling 3D objects.
Monte Carlo Neural PDE Solver for Learning PDEs via Probabilistic Representation
To address these limitations, we propose the Monte Carlo Neural PDE Solver (MCNP Solver) for training unsupervised neural solvers via the PDEs' probabilistic representation, which regards macroscopic phenomena as ensembles of random particles.
Provable Adaptivity in Adam
In particular, the existing analysis of Adam cannot clearly demonstrate the advantage of Adam over SGD.
Breaking Correlation Shift via Conditional Invariant Regularizer
The correlation shift is caused by the spurious attributes that correlate to the class label, as the correlation between them may vary in training and test data.
Deep Random Vortex Method for Simulation and Inference of Navier-Stokes Equations
To this end, we propose the \emph{Deep Random Vortex Method} (DRVM), which combines the neural network with a random vortex dynamics system equivalent to the Navier-Stokes equation.
Neural Operator with Regularity Structure for Modeling Dynamics Driven by SPDEs
Stochastic partial differential equations (SPDEs) are significant tools for modeling dynamics in many areas including atmospheric sciences and physics.
Does Momentum Change the Implicit Regularization on Separable Data?
The momentum acceleration technique is widely adopted in many optimization algorithms.
Gradient Information Matters in Policy Optimization by Back-propagating through Model
Then we proposed a two-model-based learning method to control the prediction error and the gradient error.
Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics
In this paper, to reduce the reliance on the numerical solver, we propose to enhance the supervised signal in the training of NODE.
Improved OOD Generalization via Adversarial Training and Pre-training
In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data.
Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
Inspired by adversarial training, we minimize this maximal expected loss (MMEL) and obtain a simple and interpretable closed-form solution: more attention should be paid to augmented samples with large loss values (i. e., harder examples).
BN-invariant sharpness regularizes the training model to better generalization
However, it has been pointed out that the usual definitions of sharpness, which consider either the maxima or the integral of loss over a $\delta$ ball of parameters around minima, cannot give consistent measurement for scale invariant neural networks, e. g., networks with batch normalization layer.
Characterization of Excess Risk for Locally Strongly Convex Population Risk
Our bounds underscore that with locally strongly convex population risk, the models trained by any proper iterative algorithm can generalize well, even for non-convex problems, and $d$ is large.
Dynamic of Stochastic Gradient Descent with State-Dependent Noise
Specifically, we show that the covariance of the noise of SGD in the local region of the local minima is a quadratic function of the state.
Robust Reinforcement Learning with Wasserstein Constraint
Robust Reinforcement Learning aims to find the optimal policy with some extent of robustness to environmental dynamics.
Interpreting Basis Path Set in Neural Networks
Based on basis path set, G-SGD algorithm significantly outperforms conventional SGD algorithm in optimizing neural networks.
THE EFFECT OF ADVERSARIAL TRAINING: A THEORETICAL CHARACTERIZATION
It has widely shown that adversarial training (Madry et al., 2018) is effective in defending adversarial attack empirically.
Path Space for Recurrent Neural Networks with ReLU Activations
Optimization algorithms like stochastic gradient descent that optimize the neural networks in the vector space of weights, which are not positively scale-invariant.
Off-policy Learning for Multiple Loggers
To make full use of such historical data, learning policies from multiple loggers becomes necessary.
G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Optimization on Multiple Manifolds
Optimization on manifold has been widely used in machine learning, to handle optimization problems with constraint.
Positively Scale-Invariant Flatness of ReLU Neural Networks
That is to say, the minimum with balanced values of basis paths will more likely to be flatter and generalize better.
Target Transfer Q-Learning and Its Convergence Analysis
In this paper, we propose to transfer the Q-function learned in the source task to the target of the Q-learning in the new task when certain safe conditions are satisfied.
Finite Sample Analysis of the GTD Policy Evaluation Algorithms in Markov Setting
(2) The convergence rate is determined by the step size, with the mixing time of the Markov process as the coefficient.
Differential Equations for Modeling Asynchronous Algorithms
Then we conduct theoretical analysis on the convergence rates of ASGD algorithm based on the continuous approximation.
$\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Convergence Analysis of Distributed Stochastic Gradient Descent with Shuffling
First, we give a mathematical formulation for the practical data processing procedure in distributed machine learning, which we call data partition with global/local shuffling.
A Communication-Efficient Parallel Algorithm for Decision Tree
After partitioning the training data onto a number of (e. g., $M$) machines, this algorithm performs both local voting and global voting in each iteration.
Asynchronous Stochastic Gradient Descent with Delay Compensation
We propose a novel technology to compensate this delay, so as to make the optimization behavior of ASGD closer to that of sequential SGD.
Generalization Error Bounds for Optimization Algorithms via Stability
Many machine learning tasks can be formulated as Regularized Empirical Risk Minimization (R-ERM), and solved by optimization algorithms such as gradient descent (GD), stochastic gradient descent (SGD), and stochastic variance reduction (SVRG).
Asynchronous Stochastic Proximal Optimization Algorithms with Variance Reduction
The results verified our theoretical findings and demonstrated the practical efficiency of the asynchronous stochastic proximal algorithms with variance reduction.
Two-Layer Generalization Analysis for Ranking Using Rademacher Average
sampling of queries and the conditional i. i. d sampling of documents per query.
Ranking Measures and Loss Functions in Learning to Rank
We show that these loss functions are upper bounds of the measure-based ranking errors.
