Search Results for author:
Found 38 papers, 4 papers with code
Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference
GIC trains a spurious attribute classifier based on two key properties of spurious correlations: (1) high correlation between spurious attributes and true labels, and (2) variability in this correlation between datasets with different group distributions.
The Dog Walking Theory: Rethinking Convergence in Federated Learning
In this paper, we study the convergence of FL on non-IID data and propose a novel \emph{Dog Walking Theory} to formulate and identify the missing element in existing research.
What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks
Specifically, we designed a novel set of sequence learning tasks to systematically evaluate and comprehend how the depth of transformer affects its ability to perform memorization, reasoning, generalization, and contextual generalization.
On the Benefits of Over-parameterization for Out-of-Distribution Generalization
We demonstrate that in this scenario, further increasing the model's parameterization can significantly reduce the OOD loss.
Improving Implicit Regularization of SGD with Preconditioning for Least Square Problems
In this paper, we study the generalization performance of SGD with preconditioning for the least squared problem.
An Improved Analysis of Langevin Algorithms with Prior Diffusion for Non-Log-Concave Sampling
Along this line, Freund et al. (2022) suggest that the modified Langevin algorithm with prior diffusion is able to converge dimension independently for strongly log-concave target distributions.
Towards Robust Graph Incremental Learning on Evolving Graphs
We provide a formal formulation and analysis of the problem, and propose a novel regularization-based technique called Structural-Shift-Risk-Mitigation (SSRM) to mitigate the impact of the structural shift on catastrophic forgetting of the inductive NGIL problem.
PRES: Toward Scalable Memory-Based Dynamic Graph Neural Networks
Memory-based Dynamic Graph Neural Networks (MDGNNs) are a family of dynamic graph neural networks that leverage a memory module to extract, distill, and memorize long-term temporal dependencies, leading to superior performance compared to memory-less counterparts.
Faster Sampling without Isoperimetry via Diffusion-based Monte Carlo
Specifically, DMC follows the reverse SDE of a diffusion process that transforms the target distribution to the standard Gaussian, utilizing a non-parametric score estimation.
Benign Oscillation of Stochastic Gradient Descent with Large Learning Rates
In view of this finding, we call such a phenomenon "benign oscillation".
How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities, enabling them to solve unseen tasks solely based on input contexts without adjusting model parameters.
Less is More: On the Feature Redundancy of Pretrained Models When Transferring to Few-shot Tasks
Transferring a pretrained model to a downstream task can be as easy as conducting linear probing with target data, that is, training a linear classifier upon frozen features extracted from the pretrained model.
Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Data
Modern deep learning models are usually highly over-parameterized so that they can overfit the training data.
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
We show that when learning a linear model with batch normalization for binary classification, gradient descent converges to a uniform margin classifier on the training data with an $\exp(-\Omega(\log^2 t))$ convergence rate.
Per-Example Gradient Regularization Improves Learning Signals from Noisy Data
In this paper, we explore the per-example gradient regularization (PEGR) and present a theoretical analysis that demonstrates its effectiveness in improving both test error and robustness against noise perturbations.
The Benefits of Mixup for Feature Learning
We consider a feature-noise data model and show that Mixup training can effectively learn the rare features (appearing in a small fraction of data) from its mixture with the common features (appearing in a large fraction of data).
Finite-Sample Analysis of Learning High-Dimensional Single ReLU Neuron
On the other hand, we provide some negative results for stochastic gradient descent (SGD) for ReLU regression with symmetric Bernoulli data: if the model is well-specified, the excess risk of SGD is provably no better than that of GLM-tron ignoring constant factors, for each problem instance; and in the noiseless case, GLM-tron can achieve a small risk while SGD unavoidably suffers from a constant risk in expectation.
The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift
Our bounds suggest that for a large class of linear regression instances, transfer learning with $O(N^2)$ source data (and scarce or no target data) is as effective as supervised learning with $N$ target data.
Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regime
Stochastic gradient descent (SGD) has achieved great success due to its superior performance in both optimization and generalization.
Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regression
In this paper, we provide a problem-dependent analysis on the last iterate risk bounds of SGD with decaying stepsize, for (overparameterized) linear regression problems.
Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization.
The Benefits of Implicit Regularization from SGD in Least Squares Problems
Stochastic gradient descent (SGD) exhibits strong algorithmic regularization effects in practice, which has been hypothesized to play an important role in the generalization of modern machine learning approaches.
Self-training Converts Weak Learners to Strong Learners in Mixture Models
We show that there exists a universal constant $C_{\mathrm{err}}>0$ such that if a pseudolabeler $\boldsymbol{\beta}_{\mathrm{pl}}$ can achieve classification error at most $C_{\mathrm{err}}$, then for any $\varepsilon>0$, an iterative self-training algorithm initialized at $\boldsymbol{\beta}_0 := \boldsymbol{\beta}_{\mathrm{pl}}$ using pseudolabels $\hat y = \mathrm{sgn}(\langle \boldsymbol{\beta}_t, \mathbf{x}\rangle)$ and using at most $\tilde O(d/\varepsilon^2)$ unlabeled examples suffices to learn the Bayes-optimal classifier up to $\varepsilon$ error, where $d$ is the ambient dimension.
Provable Robustness of Adversarial Training for Learning Halfspaces with Noise
To the best of our knowledge, this is the first work to show that adversarial training provably yields robust classifiers in the presence of noise.
Benign Overfitting of Constant-Stepsize SGD for Linear Regression
More specifically, for SGD with iterate averaging, we demonstrate the sharpness of the established excess risk bound by proving a matching lower bound (up to constant factors).
Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate
Understanding the algorithmic bias of \emph{stochastic gradient descent} (SGD) is one of the key challenges in modern machine learning and deep learning theory.
Faster Convergence of Stochastic Gradient Langevin Dynamics for Non-Log-Concave Sampling
We provide a new convergence analysis of stochastic gradient Langevin dynamics (SGLD) for sampling from a class of distributions that can be non-log-concave.
Improving Adversarial Robustness Requires Revisiting Misclassified Examples
In this paper, we investigate the distinctive influence of misclassified and correctly classified examples on the final robustness of adversarial training.
On the Global Convergence of Training Deep Linear ResNets
We further propose a modified identity input and output transformations, and show that a $(d+k)$-wide neural network is sufficient to guarantee the global convergence of GD/SGD, where $d, k$ are the input and output dimensions respectively.
Stochastic Gradient Hamiltonian Monte Carlo Methods with Recursive Variance Reduction
Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) algorithms have received increasing attention in both theory and practice.
How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the training sample size $n$ and the inverse of the target error $\epsilon^{-1}$, deep neural networks learned by (stochastic) gradient descent enjoy nice optimization and generalization guarantees.
Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks
Original full-batch GCN training requires calculating the representation of all the nodes in the graph per GCN layer, which brings in high computation and memory costs.
Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo
As an important Markov Chain Monte Carlo (MCMC) method, stochastic gradient Langevin dynamics (SGLD) algorithm has achieved great success in Bayesian learning and posterior sampling.
An Improved Analysis of Training Over-parameterized Deep Neural Networks
A recent line of research has shown that gradient-based algorithms with random initialization can converge to the global minima of the training loss for over-parameterized (i. e., sufficiently wide) deep neural networks.
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both gradient descent and stochastic gradient descent can find the global minima of the training loss for an over-parameterized deep ReLU network, under mild assumption on the training data.
Stochastic Variance-Reduced Hamilton Monte Carlo Methods
We propose a fast stochastic Hamilton Monte Carlo (HMC) method, for sampling from a smooth and strongly log-concave distribution.
Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently
We propose a family of nonconvex optimization algorithms that are able to save gradient and negative curvature computations to a large extent, and are guaranteed to find an approximate local minimum with improved runtime complexity.
Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization
Furthermore, for the first time we prove the global convergence guarantee for variance reduced stochastic gradient Langevin dynamics (SVRG-LD) to the almost minimizer within $\tilde O\big(\sqrt{n}d^5/(\lambda^4\epsilon^{5/2})\big)$ stochastic gradient evaluations, which outperforms the gradient complexities of GLD and SGLD in a wide regime.
