Search Results for author:
Found 70 papers, 10 papers with code
Advances and Open Problems in Federated Learning
FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches.
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
We obtain tight convergence rates for FedAvg and prove that it suffers from `client-drift' when the data is heterogeneous (non-iid), resulting in unstable and slow convergence.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
FedJAX: Federated learning simulation with JAX
Federated learning is a machine learning technique that enables training across decentralized data.
Agnostic Federated Learning
A key learning scenario in large-scale applications is that of federated learning, where a centralized model is trained based on data originating from a large number of clients.
Model-Powered Conditional Independence Test
We consider the problem of non-parametric Conditional Independence testing (CI testing) for continuous random variables.
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.
AdaCliP: Adaptive Clipping for Private SGD
Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed.
Three Approaches for Personalization with Applications to Federated Learning
The standard objective in machine learning is to train a single model for all users.
cpSGD: Communication-efficient and differentially-private distributed SGD
Distributed stochastic gradient descent is an important subroutine in distributed learning.
Lattice Rescoring Strategies for Long Short Term Memory Language Models in Speech Recognition
Recurrent neural network (RNN) language models (LMs) and Long Short Term Memory (LSTM) LMs, a variant of RNN LMs, have been shown to outperform traditional N-gram LMs on speech recognition tasks.
Federated Learning: Strategies for Improving Communication Efficiency
We consider learning algorithms for this setting where on each round, each client independently computes an update to the current model based on its local data, and communicates this update to a central server, where the client-side updates are aggregated to compute a new global model.
Distributed Mean Estimation with Limited Communication
Motivated by the need for distributed learning and optimization algorithms with low communication cost, we study communication efficient algorithms for distributed mean estimation.
Sample complexity of population recovery
For noisy population recovery, the sharp sample complexity turns out to be more sensitive to dimension and scales as $\exp(\Theta(d^{1/3} \log^{2/3}(1/\delta)))$ except for the trivial cases of $\epsilon=0, 1/2$ or $1$.
Maximum Selection and Ranking under Noisy Comparisons
We consider $(\epsilon,\delta)$-PAC maximum-selection and ranking for general probabilistic models whose comparisons probabilities satisfy strong stochastic transitivity and stochastic triangle inequality.
A Unified Maximum Likelihood Approach for Optimal Distribution Property Estimation
The advent of data science has spurred interest in estimating properties of distributions over large alphabets.
Orthogonal Random Features
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error.
Estimating Renyi Entropy of Discrete Distributions
It was recently shown that estimating the Shannon entropy $H({\rm p})$ of a discrete $k$-symbol distribution ${\rm p}$ requires $\Theta(k/\log k)$ samples, a number that grows near-linearly in the support size.
Estimating the number of unseen species: A bird in the hand is worth $\log n $ in the bush
We derive a class of estimators that $\textit{provably}$ predict $U$ not just for constant $t>1$, but all the way up to $t$ proportional to $\log n$.
Faster Algorithms for Testing under Conditional Sampling
There has been considerable recent interest in distribution-tests whose run-time and sample requirements are sublinear in the domain-size $k$.
Competitive Distribution Estimation
We also provide an estimator that runs in linear time and incurs competitive regret of $\tilde{\mathcal{O}}(\min(k/n, 1/\sqrt n))$, and show that for natural estimators this competitive regret is inevitable.
Sparse Solutions to Nonnegative Linear Systems and Applications
For learning a mixture of $k$ axis-aligned Gaussians in $d$ dimensions, we give an algorithm that outputs a mixture of $O(k/\epsilon^3)$ Gaussians that is $\epsilon$-close in statistical distance to the true distribution, without any separation assumptions.
Universal Compression of Envelope Classes: Tight Characterization via Poisson Sampling
The Poisson-sampling technique eliminates dependencies among symbol appearances in a random sequence.
Near-optimal-sample estimators for spherical Gaussian mixtures
For mixtures of any $k$ $d$-dimensional spherical Gaussians, we derive an intuitive spectral-estimator that uses $\mathcal{O}_k\bigl(\frac{d\log^2d}{\epsilon^4}\bigr)$ samples and runs in time $\mathcal{O}_{k,\epsilon}(d^3\log^5 d)$, both significantly lower than previously known.
WEST: Word Encoded Sequence Transducers
Most of the parameters in large vocabulary models are used in embedding layer to map categorical features to vectors and in softmax layer for classification weights.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Multiscale Quantization for Fast Similarity Search
We propose a multiscale quantization approach for fast similarity search on large, high-dimensional datasets.
Competitive Distribution Estimation: Why is Good-Turing Good
Second, they estimate every distribution nearly as well as the best estimator designed with prior knowledge of the exact distribution, but as all natural estimators, restricted to assign the same probability to all symbols appearing the same number of times. Specifically, for distributions over $k$ symbols and $n$ samples, we show that for both comparisons, a simple variant of Good-Turing estimator is always within KL divergence of $(3+o(1))/n^{1/3}$ from the best estimator, and that a more involved estimator is within $\tilde{\mathcal{O}}(\min(k/n, 1/\sqrt n))$.
A Unified Maximum Likelihood Approach for Estimating Symmetric Properties of Discrete Distributions
Symmetric distribution properties such as support size, support coverage, entropy, and proximity to uniformity, arise in many applications.
Approximating probabilistic models as weighted finite automata
Weighted finite automata (WFA) are often used to represent probabilistic models, such as $n$-gram language models, since they are efficient for recognition tasks in time and space.
Sampled Softmax with Random Fourier Features
For the settings where a large number of classes are involved, a common method to speed up training is to sample a subset of classes and utilize an estimate of the loss gradient based on these classes, known as the sampled softmax method.
Optimal multiclass overfitting by sequence reconstruction from Hamming queries
A primary concern of excessive reuse of test datasets in machine learning is that it can lead to overfitting.
Differentially private anonymized histograms
For a dataset of label-count pairs, an anonymized histogram is the multiset of counts.
Federated Learning of N-gram Language Models
The n-gram language models trained with federated learning are compared to n-grams trained with traditional server-based algorithms using A/B tests on tens of millions of users of virtual keyboard.
Can You Really Backdoor Federated Learning?
This paper focuses on backdoor attacks in the federated learning setting, where the goal of the adversary is to reduce the performance of the model on targeted tasks while maintaining good performance on the main task.
Relative Deviation Margin Bounds
We present a series of new and more favorable margin-based learning guarantees that depend on the empirical margin loss of a predictor.
A Theory of Multiple-Source Adaptation with Limited Target Labeled Data
We present a theoretical and algorithmic study of the multiple-source domain adaptation problem in the common scenario where the learner has access only to a limited amount of labeled target data, but where the learner has at disposal a large amount of labeled data from multiple source domains.
Learning discrete distributions: user vs item-level privacy
If each user has $m$ samples, we show that straightforward applications of Laplace or Gaussian mechanisms require the number of users to be $\mathcal{O}(k/(m\alpha^2) + k/\epsilon\alpha)$ to achieve an $\ell_1$ distance of $\alpha$ between the true and estimated distributions, with the privacy-induced penalty $k/\epsilon\alpha$ independent of the number of samples per user $m$.
Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs
We consider a distributed empirical risk minimization (ERM) optimization problem with communication efficiency and privacy requirements, motivated by the federated learning (FL) framework.
A Discriminative Technique for Multiple-Source Adaptation
We present a new discriminative technique for the multiple-source adaptation, MSA, problem.
FedBoost: A Communication-Efficient Algorithm for Federated Learning
We provide communication-efficient ensemble algorithms for federated learning, where per-round communication cost is independent of the size of the ensemble.
Robust hypothesis testing and distribution estimation in Hellinger distance
We propose a simple robust hypothesis test that has the same sample complexity as that of the optimal Neyman-Pearson test up to constants, but robust to distribution perturbations under Hellinger distance.
Wyner-Ziv Estimators for Distributed Mean Estimation with Side Information and Optimization
We propose \emph{Wyner-Ziv estimators}, which are communication and computationally efficient and near-optimal when an upper bound for the distance between the side information and the data is known.
Learning with User-Level Privacy
We show that for high-dimensional mean estimation, empirical risk minimization with smooth losses, stochastic convex optimization, and learning hypothesis classes with finite metric entropy, the privacy cost decreases as $O(1/\sqrt{m})$ as users provide more samples.
Remember What You Want to Forget: Algorithms for Machine Unlearning
We study the problem of unlearning datapoints from a learnt model.
Communication-Efficient Agnostic Federated Averaging
We propose a communication-efficient distributed algorithm called Agnostic Federated Averaging (or AgnosticFedAvg) to minimize the domain-agnostic objective proposed in Mohri et al. (2019), which is amenable to other private mechanisms such as secure aggregation.
On the Renyi Differential Privacy of the Shuffle Model
The central question studied in this paper is Renyi Differential Privacy (RDP) guarantees for general discrete local mechanisms in the shuffle privacy model.
On the benefits of maximum likelihood estimation for Regression and Forecasting
We also demonstrate empirically that our method instantiated with a well-designed general purpose mixture likelihood family can obtain superior performance for a variety of tasks across time-series forecasting and regression datasets with different data distributions.
HD-cos Networks: Efficient Neural Architechtures for Secure Multi-Party Computation
In this paper, we study training and inference of neural networks under the MPC setup.
HD-cos Networks: Efficient Neural Architectures for Secure Multi-Party Computation
In this paper, we study training and inference of neural networks under the MPC setup.
Robust Estimation for Random Graphs
We study the problem of robustly estimating the parameter $p$ of an Erd\H{o}s-R\'enyi random graph on $n$ nodes, where a $\gamma$ fraction of nodes may be adversarially corrupted.
Boosting with Multiple Sources
We study the problem of learning accurate ensemble predictors, in particular boosting, in the presence of multiple source domains.
Breaking the centralized barrier for cross-device federated learning
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.
The Fundamental Price of Secure Aggregation in Differentially Private Federated Learning
We consider the problem of training a $d$ dimensional model with distributed differential privacy (DP) where secure aggregation (SecAgg) is used to ensure that the server only sees the noisy sum of $n$ model updates in every training round.
Correlated quantization for distributed mean estimation and optimization
We show that applying the proposed protocol as sub-routine in distributed optimization algorithms leads to better convergence rates.
Scaling Language Model Size in Cross-Device Federated Learning
Most studies in cross-device federated learning focus on small models, due to the server-client communication and on-device computation bottlenecks.
Differentially Private Learning with Margin Guarantees
For the family of linear hypotheses, we give a pure DP learning algorithm that benefits from relative deviation margin guarantees, as well as an efficient DP learning algorithm with margin guarantees.
Algorithms for bounding contribution for histogram estimation under user-level privacy
In this scenario, the amount of noise injected into the histogram to obtain differential privacy is proportional to the maximum user contribution, which can be amplified by few outliers.
Private Domain Adaptation from a Public Source
A key problem in a variety of applications is that of domain adaptation from a public source domain, for which a relatively large amount of labeled data with no privacy constraints is at one's disposal, to a private target domain, for which a private sample is available with very few or no labeled data.
Concentration Bounds for Discrete Distribution Estimation in KL Divergence
We study the problem of discrete distribution estimation in KL divergence and provide concentration bounds for the Laplace estimator.
Subset-Based Instance Optimality in Private Estimation
We propose a new definition of instance optimality for differentially private estimation algorithms.
FedYolo: Augmenting Federated Learning with Pretrained Transformers
These insights on scale and modularity motivate a new federated learning approach we call "You Only Load Once" (FedYolo): The clients load a full PTF model once and all future updates are accomplished through communication-efficient modules with limited catastrophic-forgetting, where each task is assigned to its own module.
The importance of feature preprocessing for differentially private linear optimization
Training machine learning models with differential privacy (DP) has received increasing interest in recent years.
SpecTr: Fast Speculative Decoding via Optimal Transport
We show that the optimal draft selection algorithm (transport plan) can be computed via linear programming, whose best-known runtime is exponential in $k$.
Multi-Group Fairness Evaluation via Conditional Value-at-Risk Testing
Here, the sample complexity for estimating the worst-case performance gap across groups (e. g., the largest difference in error rates) increases exponentially with the number of group-denoting sensitive attributes.
Mean estimation in the add-remove model of differential privacy
We propose a new algorithm and show that it is min-max optimal, achieving the best possible constant in the leading term of the mean squared error for all $\epsilon$, and that this constant is the same as the optimal algorithm under the swap model.
Theoretical guarantees on the best-of-n alignment policy
A commonly used analytical expression in the literature claims that the KL divergence between the best-of-$n$ policy and the base policy is equal to $\log (n) - (n-1)/n.$ We disprove the validity of this claim, and show that it is an upper bound on the actual KL divergence.
Efficient Language Model Architectures for Differentially Private Federated Learning
Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices.
Optimal Block-Level Draft Verification for Accelerating Speculative Decoding
To the best of our knowledge, our work is the first to establish improvement over speculative decoding through a better draft verification algorithm.
Asymptotics of Language Model Alignment
The goal of language model alignment is to alter $p$ to a new distribution $\phi$ that results in a higher expected reward while keeping $\phi$ close to $p.$ A popular alignment method is the KL-constrained reinforcement learning (RL), which chooses a distribution $\phi_\Delta$ that maximizes $E_{\phi_{\Delta}} r(y)$ subject to a relative entropy constraint $KL(\phi_\Delta || p) \leq \Delta.$ Another simple alignment method is best-of-$N$, where $N$ samples are drawn from $p$ and one with highest reward is selected.
Towards Fast Inference: Exploring and Improving Blockwise Parallel Drafts
Despite the remarkable strides made by autoregressive language models, their potential is often hampered by the slow inference speeds inherent in sequential token generation.
