Search Results for author:
Found 29 papers, 12 papers with code
How Prior Probability Influences Decision Making: A Unifying Probabilistic Model
To explain this data, a second model has been proposed which assumes a time-varying influence of the prior.
Neurons as Monte Carlo Samplers: Bayesian Inference and Learning in Spiking Networks
Each spike in the population of inference neurons represents a sample of a particular hidden world state.
Learning Efficient Representations for Reinforcement Learning
Markov decision processes (MDPs) are a well studied framework for solving sequential decision making problems under uncertainty.
Partitioning Large Scale Deep Belief Networks Using Dropout
Deep learning methods have shown great promise in many practical applications, ranging from speech recognition, visual object recognition, to text processing.
Regularized Evolution for Image Classifier Architecture Search
The effort devoted to hand-crafting neural network image classifiers has motivated the use of architecture search to discover them automatically.
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Scaling up deep neural network capacity has been known as an effective approach to improving model quality for several different machine learning tasks.
Ranked #4 on
Fine-Grained Image Classification
on Birdsnap
(using extra training data)
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
Do CNNs Encode Data Augmentations?
To answer this question, we introduce a systematic approach to investigate which layers of neural networks are the most predictive of augmentation transformations.
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Neural network scaling has been critical for improving the model quality in many real-world machine learning applications with vast amounts of training data and compute.
Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights.
Exploring Routing Strategies for Multilingual Mixture-of-Experts Models
Sparsely-Gated Mixture-of-Experts (MoE) has been a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation.
GSPMD: General and Scalable Parallelization for ML Computation Graphs
We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations.
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
On WMT, our task-MoE with 32 experts (533M parameters) outperforms the best performing token-level MoE model (token-MoE) by +1. 0 BLEU on average across 30 language pairs.
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio.
Ranked #1 on
Speech Recognition
on Common Voice
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing.
Ranked #10 on
Language Modelling
on LAMBADA
LaMDA: Language Models for Dialog Applications
We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.
Ranked #112 on
Code Generation
on HumanEval
Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations.
ST-MoE: Designing Stable and Transferable Sparse Expert Models
But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning.
Ranked #1 on
Common Sense Reasoning
on ARC (Easy)
Mixture-of-Experts with Expert Choice Routing
Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens.
Building Machine Translation Systems for the Next Thousand Languages
In this paper we share findings from our effort to build practical machine translation (MT) systems capable of translating across over one thousand languages.
Scaling Instruction-Finetuned Language Models
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #1 on
Multi-task Language Understanding
on BBH-nlp
Massively Multilingual Shallow Fusion with Large Language Models
In this work, we propose to train a single multilingual language model (LM) for shallow fusion in multiple languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Ranked #1 on
Question Answering
on StrategyQA
Lifelong Language Pretraining with Distribution-Specialized Experts
Compared to existing lifelong learning approaches, Lifelong-MoE achieves better few-shot performance on 19 downstream NLP tasks.
Brainformers: Trading Simplicity for Efficiency
Using this insight, we develop a complex block, named Brainformer, that consists of a diverse sets of layers such as sparsely gated feed-forward layers, dense feed-forward layers, attention layers, and various forms of layer normalization and activation functions.
SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos.
Controlled Decoding from Language Models
KL-regularized reinforcement learning (RL) is a popular alignment framework to control the language model responses towards high reward outcomes.
Language Modelling
Multi-Objective Reinforcement Learning
+1
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
