READ: Recurrent Adaptation of Large Transformers

no code implementations • 24 May 2023 • Sid Wang, John Nguyen, Ke Li, Carole-Jean Wu

However, fine-tuning all pre-trained model parameters becomes impractical as the model size and number of tasks increase.

Transfer Learning

Now It Sounds Like You: Learning Personalized Vocabulary On Device

no code implementations • 5 May 2023 • Sid Wang, Ashish Shenoy, Pierce Chuang, John Nguyen

In recent years, Federated Learning (FL) has shown significant advancements in its ability to perform various natural language processing (NLP) tasks.

Federated Learning Language Modelling

TreePiece: Faster Semantic Parsing via Tree Tokenization

no code implementations • 30 Mar 2023 • Sid Wang, Akshat Shrivastava, Sasha Livshits

Autoregressive (AR) encoder-decoder neural networks have proved successful in many NLP problems, including Semantic Parsing -- a task that translates natural language to machine-readable parse trees.

Semantic Parsing

Privately Customizing Prefinetuning to Better Match User Data in Federated Learning

no code implementations • 17 Feb 2023 • Charlie Hou, Hongyuan Zhan, Akshat Shrivastava, Sid Wang, Aleksandr Livshits, Giulia Fanti, Daniel Lazar

To this end, we propose FreD (Federated Private Fr\'echet Distance) -- a privately computed distance between a prefinetuning dataset and federated datasets.

Federated Learning Language Modelling +2