Search Results for author:
Found 8 papers, 1 papers with code
Scaling Laws for Differentially Private Language Models
Scaling laws have emerged as important components of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper-parameter choices that would otherwise be expensive.
Balls-and-Bins Sampling for DP-SGD
We introduce the Balls-and-Bins sampling for differentially private (DP) optimization methods such as DP-SGD.
Scalable DP-SGD: Shuffling vs. Poisson Subsampling
We compare the utility of models trained with Poisson-subsampling-based DP-SGD, and the optimistic estimates of utility when using shuffling, via our new lower bounds on the privacy guarantee of ABLQ with shuffling.
Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models
Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora.
Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning
Large language models (LLMs) have emerged as powerful tools for tackling complex tasks across diverse domains, but they also raise privacy concerns when fine-tuned on sensitive data due to potential memorization.
How Private are DP-SGD Implementations?
We demonstrate a substantial gap between the privacy guarantees of the Adaptive Batch Linear Queries (ABLQ) mechanism under different types of batch sampling: (i) Shuffling, and (ii) Poisson subsampling; the typical analysis of Differentially Private Stochastic Gradient Descent (DP-SGD) follows by interpreting it as a post-processing of ABLQ.
Training Differentially Private Ad Prediction Models with Semi-Sensitive Features
Motivated by problems arising in digital advertising, we introduce the task of training differentially private (DP) machine learning models with semi-sensitive features.
Private Ad Modeling with DP-SGD
A well-known algorithm in privacy-preserving ML is differentially private stochastic gradient descent (DP-SGD).
