Search Results for author:
Found 37 papers, 9 papers with code
Training Private Models That Know What They Don't Know
Training reliable deep learning models which avoid making overconfident but incorrect predictions is a longstanding challenge.
Faster Differentially Private Convex Optimization via Second-Order Methods
We first develop a private variant of the regularized cubic Newton method of Nesterov and Polyak, and show that for the class of strongly convex loss functions, our algorithm has quadratic convergence and achieves the optimal excess loss.
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
However, while some adoption of DP has happened in industry, attempts to apply DP to real world complex ML models are still few and far between.
Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Risks
The first kind of oracles can estimate the gradient of one point, and the second kind of oracles, less precise and more cost-effective, can estimate the gradient difference between two points.
Why Is Public Pretraining Necessary for Private Model Training?
To explain this phenomenon, we hypothesize that the non-convex loss landscape of a model training necessitates an optimization algorithm to go through two phases.
Multi-Task Differential Privacy Under Distribution Skew
We study the problem of multi-task learning under user-level differential privacy, in which $n$ users contribute data to $m$ tasks, each involving a subset of users.
Differentially Private Image Classification from Features
We find that linear regression is much more effective than logistic regression from both privacy and computational aspects, especially at stricter epsilon values ($\epsilon < 1$).
Ranked #32 on
Image Classification
on ImageNet
Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning
We formalize the problem of DP mechanisms for adaptive streams with multiple participations and introduce a non-trivial extension of online matrix factorization DP mechanisms to our setting.
Fully Adaptive Composition for Gaussian Differential Privacy
We show that Gaussian Differential Privacy, a variant of differential privacy tailored to the analysis of Gaussian noise addition, composes gracefully even in the presence of a fully adaptive analyst.
Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Search
For instance, we achieve 77. 9% accuracy for $(\varepsilon, \delta)=(2, 10^{-5})$ on CIFAR-100 for a model pretrained on ImageNet.
Recycling Scraps: Improving Private Learning by Leveraging Intermediate Checkpoints
Empirically, we show that the last few checkpoints can provide a reasonable lower bound for the variance of a converged DP model.
(Nearly) Optimal Private Linear Regression via Adaptive Clipping
Compared to existing $(\epsilon, \delta)$-DP techniques which have sub-optimal error bounds, DP-AMBSSGD is able to provide nearly optimal error bounds in terms of key parameters like dimensionality $d$, number of points $N$, and the standard deviation $\sigma$ of the noise in observations.
Measuring Forgetting of Memorized Training Examples
In memorization, models overfit specific training examples and become susceptible to privacy attacks.
Large Scale Transfer Learning for Differentially Private Image Classification
Moreover, by systematically comparing private and non-private models across a range of large batch sizes, we find that similar to non-private setting, choice of optimizer can further improve performance substantially with DP.
Differentially Private Sampling from Rashomon Sets, and the Universality of Langevin Diffusion for Convex Optimization
In this paper we provide an algorithmic framework based on Langevin diffusion (LD) and its corresponding discretizations that allow us to simultaneously obtain: i) An algorithm for sampling from the exponential mechanism, whose privacy analysis does not depend on convexity and which can be stopped at anytime without compromising privacy, and ii) tight uniform stability guarantees for the exponential mechanism.
Toward Training at ImageNet Scale with Differential Privacy
Despite a rich literature on how to train ML models with differential privacy, it remains extremely challenging to train real-life, large neural networks with both reasonable accuracy and privacy.
Public Data-Assisted Mirror Descent for Private Model Training
In this paper, we revisit the problem of using in-distribution public data to improve the privacy/utility trade-offs for differentially private (DP) model training.
Private Alternating Least Squares: Practical Private Matrix Completion with Tighter Rates
We study the problem of differentially private (DP) matrix completion under user-level privacy.
Practical and Private (Deep) Learning without Sampling or Shuffling
We consider training models with differential privacy (DP) using mini-batch gradients.
Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning
DP formalizes this data leakage through a cryptographic game, where an adversary must predict if a model was trained on a dataset D, or a dataset D' that differs in just one example. If observing the training algorithm does not meaningfully increase the adversary's odds of successfully guessing which dataset the model was trained on, then the algorithm is said to be differentially private.
The Flajolet-Martin Sketch Itself Preserves Differential Privacy: Private Counting with Minimal Space
We propose an $(\epsilon,\delta)$-differentially private algorithm that approximates $\dist$ within a factor of $(1\pm\gamma)$, and with additive error of $O(\sqrt{\ln(1/\delta)}/\epsilon)$, using space $O(\ln(\ln(u)/\gamma)/\gamma^2)$.
Is Private Learning Possible with Instance Encoding?
A private machine learning algorithm hides as much as possible about its training data while still preserving accuracy.
Fast Dimension Independent Private AdaGrad on Publicly Estimated Subspaces
In particular, we show that if the gradients lie in a known constant rank subspace, and assuming algorithmic access to an envelope which bounds decaying sensitivity, one can achieve faster convergence to an excess empirical risk of $\tilde O(1/\epsilon n)$, where $\epsilon$ is the privacy budget and $n$ the number of samples.
Tempered Sigmoid Activations for Deep Learning with Differential Privacy
Because learning sometimes involves sensitive data, machine learning algorithms have been extended to offer privacy for training data.
Privacy Amplification via Random Check-Ins
It has privacy/accuracy trade-offs similar to privacy amplification by subsampling/shuffling.
Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descent
We show that for unconstrained convex generalized linear models (GLMs), one can obtain an excess empirical risk of $\tilde O\left(\sqrt{{\texttt{rank}}}/\epsilon n\right)$, where ${\texttt{rank}}$ is the rank of the feature matrix in the GLM problem, $n$ is the number of data samples, and $\epsilon$ is the privacy parameter.
A Separation Result Between Data-oblivious and Data-aware Poisoning Attacks
Some of the stronger poisoning attacks require the full knowledge of the training data.
Making the Shoe Fit: Architectures, Initializations, and Tuning for Learning with Privacy
Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data.
Private Stochastic Convex Optimization with Optimal Rates
A long line of existing work on private convex optimization focuses on the empirical loss and derives asymptotically tight bounds on the excess empirical loss.
Amplification by Shuffling: From Local to Central Differential Privacy via Anonymity
We study the collection of such statistics in the local differential privacy (LDP) model, and describe an algorithm whose privacy cost is polylogarithmic in the number of changes to a user's value.
Privacy Amplification by Iteration
In addition, we demonstrate that we can achieve guarantees similar to those obtainable using the privacy-amplification-by-sampling technique in several natural settings where that technique cannot be applied.
Model-Agnostic Private Learning via Stability
We provide a new technique to boost the average-case stability properties of learning algorithms to strong (worst-case) stability properties, and then exploit them to obtain private classification algorithms.
Differentially Private Matrix Completion Revisited
We provide the first provably joint differentially private algorithm with formal utility guarantees for the problem of user-level privacy-preserving collaborative filtering.
To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout
Moreover, using the above mentioned stability properties of dropout, we design dropout based differentially private algorithms for solving ERMs.
Private Empirical Risk Minimization Beyond the Worst Case: The Effect of the Constraint Set Geometry
In addition, we show that when the loss function is Lipschitz with respect to the $\ell_1$ norm and $\mathcal{C}$ is $\ell_1$-bounded, a differentially private version of the Frank-Wolfe algorithm gives error bounds of the form $\tilde{O}(n^{-2/3})$.
Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds
We provide a separate set of algorithms and matching lower bounds for the setting in which the loss functions are known to also be strongly convex.
Analyze Gauss: Optimal Bounds for Privacy-Preserving Principal Component Analysis
We show that the well-known, but misnamed, randomized response algorithm, with properly tuned parameters, provides a nearly optimal additive quality gap compared to the best possible singular subspace of A.
