Search Results for author:
Found 24 papers, 4 papers with code
Faster Algorithms for User-Level Private Stochastic Convex Optimization
Existing algorithms for user-level DP SCO are impractical in many large-scale machine learning scenarios because: (i) they make restrictive assumptions on the smoothness parameter of the loss function and require the number of users to grow polynomially with the dimension of the parameter space; or (ii) they are prohibitively slow, requiring at least $(mn)^{3/2}$ gradient computations for smooth losses and $(mn)^3$ computations for non-smooth losses.
Adaptive Batch Size for Privately Finding Second-Order Stationary Points
There is a gap between finding a first-order stationary point (FOSP) and a second-order stationary point (SOSP) under differential privacy constraints, and it remains unclear whether privately finding an SOSP is more challenging than finding an FOSP.
Improved Sample Complexity for Private Nonsmooth Nonconvex Optimization
We study differentially private (DP) optimization algorithms for stochastic and empirical objectives which are neither smooth nor convex, and propose methods that return a Goldstein-stationary point with sample complexity bounds that improve on existing works.
MUSE: Machine Unlearning Six-Way Evaluation for Language Models
Data owners may request the removal of their data from a trained model due to privacy or copyright concerns.
Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning
Large language models (LLMs) have emerged as powerful tools for tackling complex tasks across diverse domains, but they also raise privacy concerns when fine-tuned on sensitive data due to potential memorization.
Private Online Learning via Lazy Algorithms
We propose a new transformation that transforms lazy online learning algorithms into private algorithms.
Private Stochastic Convex Optimization with Heavy Tails: Near-Optimality from Simple Reductions
We study the problem of differentially private stochastic convex optimization (DP-SCO) with heavy-tailed gradients, where we assume a $k^{\text{th}}$-moment bound on the Lipschitz constants of sample functions rather than a uniform bound.
Private Gradient Descent for Linear Regression: Tighter Error Bounds and Instance-Specific Uncertainty Estimation
We provide an improved analysis of standard differentially private gradient descent for linear regression under the squared error loss.
User-level Differentially Private Stochastic Convex Optimization: Efficient Algorithms with Optimal Rates
We study differentially private stochastic convex optimization (DP-SCO) under user-level privacy, where each user may hold multiple data items.
Detecting Pretraining Data from Large Language Models
Min-K% Prob can be applied without any knowledge about the pretraining corpus or any additional training, departing from previous detection methods that require training a reference model on data that is similar to the pretraining data.
Learning across Data Owners with Joint Differential Privacy
In this paper, we study the setting in which data owners train machine learning models collaboratively under a privacy notion called joint differential privacy [Kearns et al., 2018].
$k$NN-Adapter: Efficient Domain Adaptation for Black-Box Language Models
Fine-tuning a language model on a new domain is standard practice for domain adaptation.
Algorithmic Aspects of the Log-Laplace Transform and a Non-Euclidean Proximal Sampler
The development of efficient sampling algorithms catering to non-Euclidean geometries has been a challenging endeavor, as discretization techniques which succeed in the Euclidean setting do not readily carry over to more general settings.
ReSQueing Parallel and Private Stochastic Convex Optimization
We give a parallel algorithm obtaining optimization error $\epsilon_{\text{opt}}$ with $d^{1/3}\epsilon_{\text{opt}}^{-2/3}$ gradient oracle query depth and $d^{1/3}\epsilon_{\text{opt}}^{-2/3} + \epsilon_{\text{opt}}^{-2}$ gradient queries in total, assuming access to a bounded-variance stochastic gradient estimator.
Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation
However, these data augmentation methods either potentially cause shifts in decision boundaries (representation interpolation), are not expressive enough (token replacement), or introduce too much computational overhead (augmentation with models).
Private Convex Optimization in General Norms
We propose a new framework for differentially private optimization of convex functions which are Lipschitz in an arbitrary norm $\|\cdot\|$.
When Does Differentially Private Learning Not Suffer in High Dimensions?
Large pretrained models can be privately fine-tuned to achieve performance approaching that of non-private models.
Private Convex Optimization via Exponential Mechanism
Furthermore, we show how to implement this mechanism using $\widetilde{O}(n \min(d, n))$ queries to $f_i(x)$ for the DP-SCO where $n$ is the number of samples/users and $d$ is the ambient dimension.
Better Private Algorithms for Correlation Clustering
In machine learning, correlation clustering is an important problem whose goal is to partition the individuals into groups that correlate with their pairwise similarities as much as possible.
Private Non-smooth ERM and SCO in Subquadratic Steps
We study the differentially private Empirical Risk Minimization (ERM) and Stochastic Convex Optimization (SCO) problems for non-smooth convex functions.
Tight lower bounds for Differentially Private ERM
We consider the lower bounds of differentially private ERM for general convex functions.
The Convergence Rate of SGD's Final Iterate: Analysis on Dimension Dependence
The best known lower bounds, however, are worse than the upper bounds by a factor of $\log T$.
The Power of Sampling: Dimension-free Risk Bounds in Private ERM
We further construct lower bounds, demonstrating that when gradients are full-rank, there is no separation between the constrained and unconstrained settings.
Private Non-smooth Empirical Risk Minimization and Stochastic Convex Optimization in Subquadratic Steps
More precisely, our differentially private algorithm requires $O(\frac{N^{3/2}}{d^{1/8}}+ \frac{N^2}{d})$ gradient queries for optimal excess empirical risk, which is achieved with the help of subsampling and smoothing the function via convolution.
