Search Results for author:
Found 38 papers, 16 papers with code
Towards hyperparameter-free optimization with differential privacy
Differential privacy (DP) is a privacy-preserving paradigm that protects the training data when training deep learning models.
LUME: LLM Unlearning with Multitask Evaluations
Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining.
A Hessian-informed hyperparameter optimization for differential learning rate
Differential learning rate (DLR), a technique that applies different learning rates to different model parameters, has been widely used in deep learning and achieved empirical success via its various forms.
Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate
Machine unlearning has been used to remove unwanted knowledge acquired by large language models (LLMs).
DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction
This approach led to the development of DP optimizers that have comparable performance with their non-private counterparts in fine-tuning tasks or in tasks with a small number of training parameters.
DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction
More specifically, by defining the ``frequency domain'' for both the gradient and differential privacy (DP) noise, we have developed a new component, called DOPPLER.
Gradient descent with generalized Newton's method
We propose the generalized Newton's method (GeN) -- a Hessian-informed approach that applies to any optimizer such as SGD and Adam, and covers the Newton-Raphson method as a sub-case.
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation
MAP efficiently identifies a Pareto set of scaling coefficients for merging multiple models, reflecting the trade-offs involved.
Pre-training Differentially Private Models with Limited Public Data
The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection.
Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach
In our work, we propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC, which not only offers a diminishing utility bound without inducing a constant clipping bias, but more importantly, it allows for an arbitrary choice of clipping threshold that is independent of the problem.
Zero redundancy distributed learning with differential privacy
Deep learning using large models have achieved great success in a wide range of domains.
On the accuracy and efficiency of group-wise clipping in differentially private optimization
Recent advances have substantially improved the accuracy, memory cost, and training speed of differentially private (DP) deep learning, especially on large vision and language models with millions to billions of parameters.
Tractable MCMC for Private Learning with Pure and Gaussian Differential Privacy
Posterior sampling, i. e., exponential mechanism to sample from the posterior distribution, provides $\varepsilon$-pure differential privacy (DP) guarantees and does not suffer from potentially unbounded privacy breach introduced by $(\varepsilon,\delta)$-approximate DP.
Coupling public and private gradient provably helps optimization
The success of large neural networks is crucially determined by the availability of data.
MISNN: Multiple Imputation via Semi-parametric Neural Networks
Multiple imputation (MI) has been widely applied to missing value problems in biomedical, social and econometric research, in order to avoid improper inference in the downstream data analysis.
Multiple Imputation with Neural Network Gaussian Process for High-dimensional Incomplete Data
Single imputation methods such as matrix completion methods do not adequately account for imputation uncertainty and hence would yield improper statistical inference.
Differentially Private Optimizers Can Learn Adversarially Robust Models
Machine learning models have shone in a variety of domains and attracted increasing attention from both the security and the privacy communities.
Accelerating Adversarial Perturbation by 50% with Semi-backward Propagation
Adversarial perturbation plays a significant role in the field of adversarial robustness, which solves a maximization problem over the input data.
Differentially Private Bias-Term Fine-tuning of Foundation Models
We study the problem of differentially private (DP) fine-tuning of large pre-trained models -- a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data.
Differentially Private Optimization on Large Model at Small Cost
Our implementation achieves state-of-the-art (SOTA) accuracy with very small extra cost: on GPT2 and at almost the same memory cost (<1% overhead), BK has 1. 03X the time complexity of the standard training (0. 83X training speed in practice), and 0. 61X the time complexity of the most efficient DP implementation (1. 36X training speed in practice).
CEDAR: Communication Efficient Distributed Analysis for Regressions
We provide theoretical investigation for the asymptotic properties of the proposed method for statistical inference as well as differential privacy, and evaluate its performance in simulations and real data analyses in comparison with several recently developed methods.
Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy
Large convolutional neural networks (CNN) can be difficult to train in the differentially private (DP) regime, since the optimization algorithms require a computationally expensive operation, known as the per-sample gradient clipping.
Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsity
In order to empower NAM with feature selection and improve the generalization, we propose the sparse neural additive models (SNAM) that employ the group sparsity regularization (e. g. Group LASSO), where each feature is learned by a sub-network whose trainable parameters are clustered as a group.
Multiple Imputation via Generative Adversarial Network for High-dimensional Blockwise Missing Value Problems
Missing data are present in most real world problems and need careful handling to preserve the prediction accuracy and statistical consistency in the downstream analysis.
Practical Adversarial Training with Differential Privacy for Deep Learning
In this work, we propose the practical adversarial training with differential privacy (DP-Adv), to combine the backbones from both communities and deliver robust and private models with high accuracy.
Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
Interestingly, we show a new equivalence between DP-SGD and DP-SGLD, implying that some non-Bayesian DP training naturally allows for uncertainty quantification.
Privacy Amplification via Iteration for Shuffled and Online PNSGD
We then analyze the online setting and provide a faster decaying scheme for the magnitude of the injected noise that also guarantees the convergence of privacy loss.
Accuracy, Interpretability, and Differential Privacy via Explainable Boosting
We show that adding differential privacy to Explainable Boosting Machines (EBMs), a recent method for training interpretable ML models, yields state-of-the-art accuracy while protecting privacy.
On the Convergence and Calibration of Deep Learning with Differential Privacy
Differentially private (DP) training preserves the data privacy usually at the cost of slower convergence (and thus lower accuracy), as well as more severe mis-calibration than its non-private counterpart.
Characterizing the SLOPE Trade-off: A Variational Perspective and the Donoho-Tanner Limit
Sorted l1 regularization has been incorporated into many methods for solving high-dimensional statistical estimation problems, including the SLOPE estimator in linear regression.
Efficient Designs of SLOPE Penalty Sequences in Finite Dimension
In this paper, we propose two efficient algorithms to design the possibly high-dimensional SLOPE penalty, in order to minimize the mean squared error.
Fast and Memory Efficient Differentially Private-SGD via JL Projections
Unlike previous attempts to make DP-SGD faster which work only on a subset of network architectures or use compiler techniques, we propose an algorithmic solution which works for any network in a black-box manner which is the main contribution of this paper.
FAST DIFFERENTIALLY PRIVATE-SGD VIA JL PROJECTIONS
Differentially Private-SGD (DP-SGD) of Abadi et al. (2016) and its variations are the only known algorithms for private training of large scale neural networks.
DebiNet: Debiasing Linear Models with Nonlinear Overparameterized Neural Networks
Recent years have witnessed strong empirical performance of over-parameterized neural networks on various tasks and many advances in the theory, e. g. the universal approximation and provable convergence to global minimum.
A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks
When equipped with efficient optimization algorithms, the over-parameterized neural networks have demonstrated high level of performance even though the loss function is non-convex and non-smooth.
The Complete Lasso Tradeoff Diagram
A fundamental problem in the high-dimensional regression is to understand the tradeoff between type I and type II errors or, equivalently, false discovery rate (FDR) and power in variable selection.
Statistics Theory Information Theory Information Theory Statistics Theory
Deep Learning with Gaussian Differential Privacy
Leveraging the appealing properties of $f$-differential privacy in handling composition and subsampling, this paper derives analytically tractable expressions for the privacy guarantees of both stochastic gradient descent and Adam used in training deep neural networks, without the need of developing sophisticated techniques as [3] did.
Algorithmic Analysis and Statistical Estimation of SLOPE via Approximate Message Passing
SLOPE is a relatively new convex optimization procedure for high-dimensional linear regression via the sorted l1 penalty: the larger the rank of the fitted coefficient, the larger the penalty.
