Search Results for author:
Found 28 papers, 9 papers with code
Differentially Private Training of Mixture of Experts Models
This position paper investigates the integration of Differential Privacy (DP) in the training of Mixture of Experts (MoE) models within the field of natural language processing.
TinyGSM: achieving >80% on GSM8k with small language models
Specifically for solving grade school math, the smallest model size so far required to break the 80\% barrier on the GSM8K benchmark remains to be 34B.
Ranked #58 on
Arithmetic Reasoning
on GSM8K
Classification with Partially Private Features
In this paper, we consider differentially private classification when some features are sensitive, while the rest of the features and the label are not.
Privately Aligning Language Models with Reinforcement Learning
Positioned between pre-training and user deployment, aligning large language models (LLMs) through reinforcement learning (RL) has emerged as a prevailing strategy for training instruction following-models such as ChatGPT.
Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks
Large language models have revolutionized the field of NLP by achieving state-of-the-art performance on various tasks.
Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation
Our results demonstrate that our algorithm can achieve competitive performance with strong privacy levels.
Differentially Private Synthetic Data via Foundation Model APIs 1: Images
We further demonstrate the promise of applying PE on large foundation models such as Stable Diffusion to tackle challenging private datasets with a small number of high-resolution images.
Selective Pre-training for Private Fine-tuning
Besides performance improvements, our framework also shows that with careful pre-training and private fine-tuning, smaller models can match the performance of much larger models that do not have access to private data, highlighting the promise of private learning as a tool for model compression and efficiency.
Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping
To reduce the compute time overhead of private learning, we show that \emph{per-layer clipping}, where the gradient of each neural network layer is clipped separately, allows clipping to be performed in conjunction with backpropagation in differentially private optimization.
When Does Differentially Private Learning Not Suffer in High Dimensions?
Large pretrained models can be privately fine-tuned to achieve performance approaching that of non-private models.
Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent
Differentially private stochastic gradient descent (DP-SGD) is the workhorse algorithm for recent advances in private deep learning.
Differentially Private Model Compression
Recent papers have shown that large pre-trained language models (LLMs) such as BERT, GPT-2 can be fine-tuned on private data to achieve performance comparable to non-private models for many downstream Natural Language Processing (NLP) tasks while simultaneously guaranteeing differential privacy.
Private Non-smooth ERM and SCO in Subquadratic Steps
We study the differentially private Empirical Risk Minimization (ERM) and Stochastic Convex Optimization (SCO) problems for non-smooth convex functions.
Differentially Private Fine-tuning of Language Models
For example, on the MNLI dataset we achieve an accuracy of $87. 8\%$ using RoBERTa-Large and $83. 5\%$ using RoBERTa-Base with a privacy budget of $\epsilon = 6. 7$.
Synergy: Resource Sensitive DNN Scheduling in Multi-Tenant Clusters
Unfortunately, these schedulers do not consider the impact of a job's sensitivity to allocation of CPU, memory, and storage resources.
Differentially Private n-gram Extraction
We revisit the problem of $n$-gram extraction in the differential privacy setting.
Accuracy, Interpretability, and Differential Privacy via Explainable Boosting
We show that adding differential privacy to Explainable Boosting Machines (EBMs), a recent method for training interpretable ML models, yields state-of-the-art accuracy while protecting privacy.
Private Non-smooth Empirical Risk Minimization and Stochastic Convex Optimization in Subquadratic Steps
More precisely, our differentially private algorithm requires $O(\frac{N^{3/2}}{d^{1/8}}+ \frac{N^2}{d})$ gradient queries for optimal excess empirical risk, which is achieved with the help of subsampling and smoothing the function via convolution.
Differentially Private Correlation Clustering
Correlation clustering is a widely used technique in unsupervised machine learning.
Fast and Memory Efficient Differentially Private-SGD via JL Projections
Unlike previous attempts to make DP-SGD faster which work only on a subset of network architectures or use compiler techniques, we propose an algorithmic solution which works for any network in a black-box manner which is the main contribution of this paper.
FAST DIFFERENTIALLY PRIVATE-SGD VIA JL PROJECTIONS
Differentially Private-SGD (DP-SGD) of Abadi et al. (2016) and its variations are the only known algorithms for private training of large scale neural networks.
Consistent $k$-Median: Simpler, Better and Robust
In this paper we introduce and study the online consistent $k$-clustering with outliers problem, generalizing the non-outlier version of the problem studied in [Lattanzi-Vassilvitskii, ICML17].
Differentially Private Set Union
Known algorithms for this problem proceed by collecting a subset of items from each user, taking the union of such subsets, and disclosing the items whose noisy counts fall above a certain threshold.
Privately Learning Markov Random Fields
We consider the problem of learning Markov Random Fields (including the prototypical example, the Ising model) under the constraint of differential privacy.
Locally Private Hypothesis Selection
Absent privacy constraints, this problem requires $O(\log k)$ samples from $p$, and it was recently shown that the same complexity is achievable under (central) differential privacy.
Locally Private Gaussian Estimation
We study a basic private estimation problem: each of $n$ users draws a single i. i. d.
Collecting Telemetry Data Privately
In particular, existing LDP algorithms are not suitable for repeated collection of counter data such as daily app usage statistics.
Truth and Regret in Online Scheduling
The service provider's goal is to implement a truthful online mechanism for scheduling jobs so as to maximize the social welfare of the schedule.
