Search Results for author:
Found 27 papers, 11 papers with code
SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos.
Brainformers: Trading Simplicity for Efficiency
Using this insight, we develop a complex block, named Brainformer, that consists of a diverse sets of layers such as sparsely gated feed-forward layers, dense feed-forward layers, attention layers, and various forms of layer normalization and activation functions.
Lifelong Language Pretraining with Distribution-Specialized Experts
Compared to existing lifelong learning approaches, Lifelong-MoE achieves better few-shot performance on 19 downstream NLP tasks.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Ranked #1 on
Question Answering
on TriviaQA
(using extra training data)
AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device.
Massively Multilingual Shallow Fusion with Large Language Models
In this work, we propose to train a single multilingual language model (LM) for shallow fusion in multiple languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Scaling Instruction-Finetuned Language Models
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #1 on
Coreference Resolution
on WSC
Cross-Lingual Question Answering
Multi-task Language Understanding
+1
Building Machine Translation Systems for the Next Thousand Languages
In this paper we share findings from our effort to build practical machine translation (MT) systems capable of translating across over one thousand languages.
Mixture-of-Experts with Expert Choice Routing
Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens.
ST-MoE: Designing Stable and Transferable Sparse Expert Models
But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning.
Ranked #1 on
Common Sense Reasoning
on ARC (Easy)
Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations.
LaMDA: Language Models for Dialog Applications
We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.
Ranked #36 on
Code Generation
on HumanEval
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing.
Ranked #9 on
Question Answering
on TriviaQA
(using extra training data)
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
On WMT, our task-MoE with 32 experts (533M parameters) outperforms the best performing token-level MoE model (token-MoE) by +1. 0 BLEU on average across 30 language pairs.
GSPMD: General and Scalable Parallelization for ML Computation Graphs
We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations.
Exploring Routing Strategies for Multilingual Mixture-of-Experts Models
Sparsely-Gated Mixture-of-Experts (MoE) has been a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation.
Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights.
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Neural network scaling has been critical for improving the model quality in many real-world machine learning applications with vast amounts of training data and compute.
Do CNNs Encode Data Augmentations?
To answer this question, we introduce a systematic approach to investigate which layers of neural networks are the most predictive of augmentation transformations.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Scaling up deep neural network capacity has been known as an effective approach to improving model quality for several different machine learning tasks.
Ranked #4 on
Fine-Grained Image Classification
on Birdsnap
(using extra training data)
Regularized Evolution for Image Classifier Architecture Search
The effort devoted to hand-crafting neural network image classifiers has motivated the use of architecture search to discover them automatically.
Learning Efficient Representations for Reinforcement Learning
Markov decision processes (MDPs) are a well studied framework for solving sequential decision making problems under uncertainty.
Partitioning Large Scale Deep Belief Networks Using Dropout
Deep learning methods have shown great promise in many practical applications, ranging from speech recognition, visual object recognition, to text processing.
Neurons as Monte Carlo Samplers: Bayesian Inference and Learning in Spiking Networks
Each spike in the population of inference neurons represents a sample of a particular hidden world state.
How Prior Probability Influences Decision Making: A Unifying Probabilistic Model
To explain this data, a second model has been proposed which assumes a time-varying influence of the prior.
