Search Results for author:
Found 26 papers, 5 papers with code
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates.
Continuous-time Riemannian SGD and SVRG Flows on Wasserstein Probabilistic Space
In this paper, we aim to enrich the continuous optimization methods in the Wasserstein space by extending the gradient flow into the stochastic gradient descent (SGD) flow and stochastic variance reduction gradient (SVRG) flow.
Large Catapults in Momentum Gradient Descent with Warmup: An Empirical Study
Although gradient descent with momentum is widely used in modern deep learning, a concrete understanding of its effects on the training trajectory still remains elusive.
Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity
Recently, Arjevani et al. [1] established a lower bound of iteration complexity for the first-order optimization under an $L$-smooth condition and a bounded noise variance assumption.
How Can Large Language Models Help Humans in Design and Manufacturing?
The advancement of Large Language Models (LLMs), including GPT-4, provides exciting new opportunities for generative design.
Fast Conditional Mixing of MCMC Algorithms for Non-log-concave Distributions
However, on the theory side, MCMC algorithms suffer from slow mixing rate when $\pi(x)$ is non-log-concave.
When and Why Momentum Accelerates SGD:An Empirical Study
In the comparison of SGDM and SGD with the same effective learning rate and the same batch size, we observe a consistent pattern: when $\eta_{ef}$ is small, SGDM and SGD experience almost the same empirical training losses; when $\eta_{ef}$ surpasses a certain threshold, SGDM begins to perform better than SGD.
ALO-VC: Any-to-any Low-latency One-shot Voice Conversion
This paper presents ALO-VC, a non-parallel low-latency one-shot phonetic posteriorgrams (PPGs) based voice conversion method.
Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions
We provide a simple convergence proof for AdaGrad optimizing non-convex objectives under only affine noise variance and bounded smoothness assumptions.
On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training
Pre-training datasets are critical for building state-of-the-art machine learning models, motivating rigorous study on their impact on downstream tasks.
O-GNN: Incorporating Ring Priors into Molecular Modeling
Despite the recent success of molecular modeling with graph neural networks (GNNs), few models explicitly take rings in compounds into consideration, consequently limiting the expressiveness of the models.
Ranked #1 on
Graph Regression
on PCQM4M-LSC
(Validation MAE metric)
InfraDet3D: Multi-Modal 3D Object Detection based on Roadside Infrastructure Camera and LiDAR Sensors
We evaluate our results on the A9 infrastructure dataset and achieve 68. 48 mAP on the test set.
Regularization of polynomial networks for image recognition
We introduce a class of PNs, which are able to reach the performance of ResNet across a range of six benchmarks.
DiGress: Discrete Denoising diffusion for graph generation
This work introduces DiGress, a discrete denoising diffusion model for generating graphs with categorical node and edge attributes.
Provable Adaptivity in Adam
In particular, the existing analysis of Adam cannot clearly demonstrate the advantage of Adam over SGD.
See More for Scene: Pairwise Consistency Learning for Scene Classification
In this paper, we propose to understand scene images and the scene classification CNN models in terms of the focus area.
Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
Does Momentum Change the Implicit Regularization on Separable Data?
The momentum acceleration technique is widely adopted in many optimization algorithms.
Creating Training Sets via Weak Indirect Supervision
Creating labeled training sets has become one of the major roadblocks in machine learning.
Machine-Learning Non-Conservative Dynamics for New-Physics Detection
Energy conservation is a basic physics principle, the breakdown of which often implies new physics.
Optimizing Information-theoretical Generalization Bound via Anisotropic Noise of SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
End-to-End Human Object Interaction Detection with HOI Transformer
We propose HOI Transformer to tackle human object interaction (HOI) detection in an end-to-end manner.
Ranked #30 on
Human-Object Interaction Detection
on HICO-DET
(using extra training data)
Robustness, Privacy, and Generalization of Adversarial Training
This measure can be approximate empirically by an asymptotically consistent empirical estimator, {\it empirical robustified intensity}.
The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
Except GD, adaptive algorithms such as AdaGrad, RMSProp and Adam are popular owing to their rapid training process.
Tighter Generalization Bounds for Iterative Differentially Private Learning Algorithms
This paper studies the relationship between generalization and privacy preservation in iterative learning algorithms by two sequential steps.
Piecewise linear activations substantially shape the loss surfaces of neural networks
This result holds for any neural network with arbitrary depth and arbitrary piecewise linear activation functions (excluding linear functions) under most loss functions in practice.
