Search Results for author:
Found 19 papers, 14 papers with code
PaLM: Scaling Language Modeling with Pathways
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
Ranked #1 on
Natural Language Inference
on RTE
Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves.
Self-Consistency Improves Chain of Thought Reasoning in Language Models
We explore a simple ensemble strategy, self-consistency, that significantly improves the reasoning accuracy of large language models.
Scale Efficiently: Insights from Pretraining and Finetuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
ByT5: Towards a token-free future with pre-trained byte-to-byte models
Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units.
Ranked #1 on
Question Answering
on TweetQA
Cross-Lingual Natural Language Inference
Cross-Lingual NER
+3
Do Transformer Modifications Transfer Across Implementations and Applications?
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption.
On Task-Level Dialogue Composition of Generative Transformer Model
In this work, we begin by studying the effect of training human-human task-oriented dialogues towards improving the ability to compose multiple tasks on Transformer generative models.
WT5?! Training Text-to-Text Models to Explain their Predictions
Neural networks have recently achieved human-level performance on various challenging natural language processing (NLP) tasks, but it is notoriously difficult to understand why a neural network produced a particular prediction.
Neural Assistant: Joint Action Prediction, Response Generation, and Latent Knowledge Reasoning
In this paper, we develop Neural Assistant: a single neural network model that takes conversation history and an external knowledge source as input and jointly produces both text response and action to be taken by the system as output.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP).
Ranked #1 on
Natural Language Inference
on MultiNLI
A Proposed Hierarchy of Deep Learning Tasks
As the pace of deep learning innovation accelerates, it becomes increasingly important to organize the space of problems by relative difficultly.
Deep Learning Scaling is Predictable, Empirically
As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art.
Block-Sparse Recurrent Neural Networks
Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms.
Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
We present Deep Voice 3, a fully-convolutional attention-based neural text-to-speech (TTS) system.
Mixed Precision Training
Using this approach, we can reduce the memory consumption of deep learning models by nearly 2x.
Exploring Sparsity in Recurrent Neural Networks
Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around 2x to 7x.
DSD: Dense-Sparse-Dense Training for Deep Neural Networks
We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance.
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech--two vastly different languages.
