Search Results for author:
Found 55 papers, 17 papers with code
©Plug-in Authorization for Human Content Copyright Protection in Text-to-Image Model
To mitigate this, we propose the \copyright Plug-in Authorization framework, introducing three operations: addition, extraction, and combination.
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates.
Differentially Private Synthetic Data via Foundation Model APIs 2: Text
Lin et al. (2024) recently introduced the Private Evolution (PE) algorithm to generate DP synthetic images with only API access to diffusion models.
Exploring Transferability for Randomized Smoothing
However, the model robustness, which is a critical aspect for safety, is often optimized for each specific task rather than at the pretraining stage.
Large Catapults in Momentum Gradient Descent with Warmup: An Empirical Study
Although gradient descent with momentum is widely used in modern deep learning, a concrete understanding of its effects on the training trajectory still remains elusive.
On the Generalization Properties of Diffusion Models
This precisely elucidates the adverse effect of "modes shift" in ground truths on the model generalization.
Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity
Recently, Arjevani et al. [1] established a lower bound of iteration complexity for the first-order optimization under an $L$-smooth condition and a bounded noise variance assumption.
FILM: How can Few-Shot Image Classification Benefit from Pre-Trained Language Models?
Few-shot learning aims to train models that can be generalized to novel classes with only a few samples.
When and Why Momentum Accelerates SGD:An Empirical Study
In the comparison of SGDM and SGD with the same effective learning rate and the same batch size, we observe a consistent pattern: when $\eta_{ef}$ is small, SGDM and SGD experience almost the same empirical training losses; when $\eta_{ef}$ surpasses a certain threshold, SGDM begins to perform better than SGD.
UADB: Unsupervised Anomaly Detection Booster
This is a challenging task given the heterogeneous model structures and assumptions adopted by existing UAD methods.
Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions
We provide a simple convergence proof for AdaGrad optimizing non-convex objectives under only affine noise variance and bounded smoothness assumptions.
Selective Pre-training for Private Fine-tuning
Besides performance improvements, our framework also shows that with careful pre-training and private fine-tuning, smaller models can match the performance of much larger models that do not have access to private data, highlighting the promise of private learning as a tool for model compression and efficiency.
ResiDual: Transformer with Dual Residual Connections
In this paper, we propose ResiDual, a novel Transformer architecture with Pre-Post-LN (PPLN), which fuses the connections in Post-LN and Pre-LN together and inherits their advantages while avoids their limitations.
Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping
To reduce the compute time overhead of private learning, we show that \emph{per-layer clipping}, where the gradient of each neural network layer is clipped separately, allows clipping to be performed in conjunction with backpropagation in differentially private optimization.
Denoising Masked AutoEncoders Help Robust Classification
In this paper, we propose a new self-supervised method, which is called Denoising Masked AutoEncoders (DMAE), for learning certified robust classifiers of images.
Provable Adaptivity in Adam
In particular, the existing analysis of Adam cannot clearly demonstrate the advantage of Adam over SGD.
Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization
By ensuring differential privacy in the learning algorithms, one can rigorously mitigate the risk of large models memorizing sensitive training data.
Adversarial Noises Are Linearly Separable for (Nearly) Random Neural Networks
Adversarial examples, which are usually generated for specific inputs with a specific model, are ubiquitous for neural networks.
Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent
Differentially private stochastic gradient descent (DP-SGD) is the workhorse algorithm for recent advances in private deep learning.
Robust Quantity-Aware Aggregation for Federated Learning
Federated learning (FL) enables multiple clients to collaboratively train models without sharing their local data, and becomes an important privacy-preserving machine learning framework.
Availability Attacks Create Shortcuts
We are the first to unveil an important population property of the perturbations of these attacks: they are almost \textbf{linearly separable} when assigned with the target labels of the corresponding samples, which hence can work as \emph{shortcuts} for the learning objective.
Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
Differentially Private Fine-tuning of Language Models
For example, on the MNLI dataset we achieve an accuracy of $87. 8\%$ using RoBERTa-Large and $83. 5\%$ using RoBERTa-Base with a privacy budget of $\epsilon = 6. 7$.
Does Momentum Change the Implicit Regularization on Separable Data?
The momentum acceleration technique is widely adopted in many optimization algorithms.
Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum
Specifically, we disentangle the effects of Adaptive Learning Rate and Momentum of the Adam dynamics on saddle-point escaping and flat minima selection.
Regularized-OFU: an efficient algorithm for general contextual bandit with optimization oracles
In contextual bandit, one major challenge is to develop theoretically solid and empirically efficient algorithms for general function classes.
Regularized OFU: an Efficient UCB Estimator forNon-linear Contextual Bandit
However, it is in general unknown how to deriveefficient and effective EE trade-off methods for non-linearcomplex tasks, suchas contextual bandit with deep neural network as the reward function.
Large Scale Private Learning via Low-rank Reparametrization
We propose a reparametrization scheme to address the challenges of applying differentially private SGD on large neural networks, which are 1) the huge memory cost of storing individual gradients, 2) the added noise suffering notorious dimensional dependence.
Two Coupled Rejection Metrics Can Tell Adversarial Examples Apart
Along with this routine, we find that confidence and a rectified confidence (R-Con) can form two coupled rejection metrics, which could provably distinguish wrongly classified inputs from correctly classified ones.
Optimizing Information-theoretical Generalization Bound via Anisotropic Noise of SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
The privacy leakage of the model about the training data can be bounded in the differential privacy mechanism.
BN-invariant sharpness regularizes the training model to better generalization
However, it has been pointed out that the usual definitions of sharpness, which consider either the maxima or the integral of loss over a $\delta$ ball of parameters around minima, cannot give consistent measurement for scale invariant neural networks, e. g., networks with batch normalization layer.
On the Stability of Multi-branch Network
More importantly, we propose a new design ``STAM aggregation" that can guarantee to STAbilize the forward/backward process of Multi-branch networks irrespective of the number of branches.
How Does Data Augmentation Affect Privacy in Machine Learning?
Even further, we show that the proposed approach can achieve higher MI attack success rates on models trained with some data augmentation than the existing methods on models trained without data augmentation.
Adai: Separating the Effects of Adaptive Learning Rate and Momentum Inertia
Specifically, we disentangle the effects of Adaptive Learning Rate and Momentum of the Adam dynamics on saddle-point escaping and minima selection.
On Layer Normalization in the Transformer Architecture
This motivates us to remove the warm-up stage for the training of Pre-LN Transformers.
Gradient Perturbation is Underrated for Differentially Private Convex Optimization
By using the \emph{expected curvature}, we show that gradient perturbation can achieve a significantly improved utility guarantee that can theoretically justify the advantage of gradient perturbation over other perturbation methods.
STABILITY AND CONVERGENCE THEORY FOR LEARNING RESNET: A FULL CHARACTERIZATION
We show that for standard initialization used in practice, $\tau =1/\Omega(\sqrt{L})$ is a sharp value in characterizing the stability of forward/backward process of ResNet, where $L$ is the number of residual blocks.
THE EFFECT OF ADVERSARIAL TRAINING: A THEORETICAL CHARACTERIZATION
It has widely shown that adversarial training (Madry et al., 2018) is effective in defending adversarial attack empirically.
Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data
Our theory captures how the convergence of distributed algorithms behaves as the number of machines and the size of local data vary.
Optimization on Multiple Manifolds
Optimization on manifold has been widely used in machine learning, to handle optimization problems with constraint.
G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Stabilize Deep ResNet with A Sharp Scaling Factor $τ$
Moreover, for ResNets with normalization layer, adding such a factor $\tau$ also stabilizes the training and obtains significant performance gain for deep ResNet.
SGD Converges to Global Minimum in Deep Learning via Star-convex Path
Stochastic gradient descent (SGD) has been found to be surprisingly effective in training a variety of deep neural networks.
On the Local Hessian in Back-propagation
We study the Hessian of the local back-matching loss (local Hessian) and connect it to the efficiency of BP.
Capacity Control of ReLU Neural Networks by Basis-path Norm
Motivated by this, we propose a new norm \emph{Basis-path Norm} based on a group of linearly independent paths to measure the capacity of neural networks more accurately.
Train Feedfoward Neural Network with Layer-wise Adaptive Rate via Approximating Back-matching Propagation
This inconsistence of gradient magnitude across different layers renders optimization of deep neural network with a single learning rate problematic.
Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization
With strongly convex regularizers, we further establish the generalization error bounds for nonconvex loss functions under proximal SGD with high-probability guarantee, i. e., exponential concentration in probability.
$\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Block-diagonal Hessian-free Optimization for Training Neural Networks
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence.
Nonconvex Low-Rank Matrix Recovery with Arbitrary Outliers via Median-Truncated Gradient Descent
Recent work has demonstrated the effectiveness of gradient descent for directly recovering the factors of low-rank matrices from random linear measurements in a globally convergent manner when initialized properly.
Reshaped Wirtinger Flow for Solving Quadratic System of Equations
In contrast to the smooth loss function used in WF, we adopt a nonsmooth but lower-order loss function, and design a gradient-like algorithm (referred to as reshaped-WF).
Reshaped Wirtinger Flow and Incremental Algorithm for Solving Quadratic System of Equations
We further develop the incremental (stochastic) reshaped Wirtinger flow (IRWF) and show that IRWF converges linearly to the true signal.
Median-Truncated Nonconvex Approach for Phase Retrieval with Outliers
This paper investigates the phase retrieval problem, which aims to recover a signal from the magnitudes of its linear measurements.
Analysis of Robust PCA via Local Incoherence
We investigate the robust PCA problem of decomposing an observed matrix into the sum of a low-rank and a sparse error matrices via convex programming Principal Component Pursuit (PCP).
