Search Results for author:
Found 15 papers, 10 papers with code
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
In particular, we focus on text-to-text models and experiment with three model architectures (causal/non-causal decoder-only and encoder-decoder), trained with two different pretraining objectives (autoregressive and masked language modeling), and evaluated with and without multitask prompted finetuning.
PaLM: Scaling Language Modeling with Pathways
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
Ranked #1 on
Natural Language Inference
on RTE
Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves.
Learning Compact Metrics for MT
Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT.
Scale Efficiently: Insights from Pretraining and Finetuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model.
Ranked #2 on
Paraphrase Identification
on Quora Question Pairs
A Simple and Effective Positional Encoding for Transformers
Our analysis shows that the gain actually comes from moving positional information to attention layer from the input.
Do Transformer Modifications Transfer Across Implementations and Applications?
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption.
Neural Data Augmentation via Example Extrapolation
In many applications of machine learning, certain categories of examples may be underrepresented in the training data, causing systems to underperform on such "few-shot" cases at test time.
Rethinking embedding coupling in pre-trained language models
We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art pre-trained language models.
Ranked #1 on
Cross-Lingual NER
on NER
Cross-Lingual Natural Language Inference
Cross-Lingual NER
+4
Improving Multilingual Models with Language-Clustered Vocabularies
State-of-the-art multilingual models depend on vocabularies that cover all of the languages the model will expect to see at inference time, but the standard methods for generating those vocabularies are not ideal for massively multilingual applications.
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task
The quality of machine translation systems has dramatically improved over the last decade, and as a result, evaluation has become an increasingly challenging problem.
Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition
Transformer-based models have achieved stateof-the-art results in many tasks in natural language processing.
Adversarial Attacks Against Medical Deep Learning Systems
The discovery of adversarial examples has raised concerns about the practical deployment of deep learning systems.
