Search Results for author:
Found 30 papers, 5 papers with code
Understanding Unintended Memorization in Language Models Under Federated Learning
Thus, we initiate a formal study to understand the effect of different components of FL on unintended memorization in trained NWP models.
Learning from straggler clients in federated learning
How well do existing federated learning algorithms learn from client devices that return model updates with a significant time delay?
Unintended Memorization in Large ASR Models, and How to Mitigate It
We empirically show that clipping each example's gradient can mitigate memorization for sped-up training examples with up to 16 repetitions in the training set.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Heterogeneous Federated Learning Using Knowledge Codistillation
Federated Averaging, and many federated learning algorithm variants which build upon it, have a limitation: all clients must share the same model architecture.
The Gift of Feedback: Improving ASR Model Quality by Learning from User Corrections through Federated Learning
Automatic speech recognition (ASR) models are typically trained on large datasets of transcribed speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Recycling Scraps: Improving Private Learning by Leveraging Intermediate Checkpoints
Empirically, we show that the last few checkpoints can provide a reasonable lower bound for the variance of a converged DP model.
Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learning
Almost none of the 2, 000+ languages spoken in Africa have widely available automatic speech recognition systems, and the required data is also only available for a few languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
UserLibri: A Dataset for ASR Personalization Using Only Text
We experiment on a user-clustered LibriSpeech corpus, supplemented with personalized text-only data for each user from Project Gutenberg.
Mixed Federated Learning: Joint Decentralized and Centralized Learning
For example, additional datacenter data can be leveraged to jointly learn from centralized (datacenter) and decentralized (federated) training data and better match an expected inference data distribution.
Online Model Compression for Federated Learning with Large Models
This paper addresses the challenges of training large neural network models under federated learning settings: high on-device memory usage and communication cost.
Detecting Unintended Memorization in Language-Model-Fused ASR
End-to-end (E2E) models are often being accompanied by language models (LMs) via shallow fusion for boosting their overall quality as well as recognition of rare words.
Extracting Targeted Training Data from ASR Models, and How to Mitigate It
We design Noise Masking, a fill-in-the-blank style method for extracting targeted parts of training data from trained ASR models.
Production federated keyword spotting via distillation, filtering, and joint federated-centralized training
We trained a keyword spotting model using federated learning on real user devices and observed significant improvements when the model was deployed for inference on phones.
Scaling Language Model Size in Cross-Device Federated Learning
Most studies in cross-device federated learning focus on small models, due to the server-client communication and on-device computation bottlenecks.
Capitalization Normalization for Language Modeling with an Accurate and Efficient Hierarchical RNN Model
Capitalization normalization (truecasing) is the task of restoring the correct case (uppercase or lowercase) of noisy text.
Public Data-Assisted Mirror Descent for Private Model Training
In this paper, we revisit the problem of using in-distribution public data to improve the privacy/utility trade-offs for differentially private (DP) model training.
Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift
With privacy as a motivation, Federated Learning (FL) is an increasingly used paradigm where learning takes place collectively on edge devices, each with a cache of user-generated training examples that remain resident on the local device.
Revealing and Protecting Labels in Distributed Training
Prior works have demonstrated that labels can be revealed analytically from the last layer of certain models (e. g., ResNet), or they can be reconstructed jointly with model inputs by using Gradients Matching [Zhu et al'19] with additional knowledge about the current state of the model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Position-Invariant Truecasing with a Word-and-Character Hierarchical Recurrent Neural Network
Truecasing is the task of restoring the correct case (uppercase or lowercase) of noisy text generated either by an automatic system for speech recognition or machine translation or by humans.
A Method to Reveal Speaker Identity in Distributed ASR Training, and How to Counter It
We show that a dropout rate of 0. 2 can reduce the speaker identity accuracy to 0% top-1 (0. 5% top-5).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Communication-Efficient Agnostic Federated Averaging
We propose a communication-efficient distributed algorithm called Agnostic Federated Averaging (or AgnosticFedAvg) to minimize the domain-agnostic objective proposed in Mohri et al. (2019), which is amenable to other private mechanisms such as secure aggregation.
Training Production Language Models without Memorizing User Data
This paper presents the first consumer-scale next-word prediction (NWP) model trained with Federated Learning (FL) while leveraging the Differentially Private Federated Averaging (DP-FedAvg) technique.
Understanding Unintended Memorization in Federated Learning
In this paper, we initiate a formal study to understand the effect of different components of canonical FL on unintended memorization in trained models, comparing with the central learning setting.
Training Keyword Spotting Models on Non-IID Data with Federated Learning
We demonstrate that a production-quality keyword-spotting model can be trained on-device using federated learning and achieve comparable false accept and false reject rates to a centrally-trained model.
Generative Models for Effective ML on Private, Decentralized Datasets
To improve real-world applications of machine learning, experienced modelers develop intuition about their datasets, their models, and how the two interact.
Federated Evaluation of On-device Personalization
Federated learning is a distributed, on-device computation framework that enables training global models without exporting sensitive user data to servers.
Federated Learning of N-gram Language Models
The n-gram language models trained with federated learning are compared to n-grams trained with traditional server-based algorithms using A/B tests on tens of millions of users of virtual keyboard.
Federated Learning for Emoji Prediction in a Mobile Keyboard
We show that a word-level recurrent neural network can predict emoji from text typed on a mobile keyboard.
Federated Learning Of Out-Of-Vocabulary Words
We demonstrate that a character-level recurrent neural network is able to learn out-of-vocabulary (OOV) words under federated learning settings, for the purpose of expanding the vocabulary of a virtual keyboard for smartphones without exporting sensitive text to servers.
Federated Learning for Mobile Keyboard Prediction
We train a recurrent neural network language model using a distributed, on-device learning framework called federated learning for the purpose of next-word prediction in a virtual keyboard for smartphones.
