Search Results for author:
Found 42 papers, 23 papers with code
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains.
OPT: Open Pre-trained Transformer Language Models
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning.
Ranked #2 on
Stereotypical Bias Analysis
on CrowS-Pairs
Efficient Language Modeling with Sparse all-MLP
All-MLP architectures have attracted increasing interest as an alternative to attention-based models.
Ranked #17 on
Question Answering
on StoryCloze
Efficient Large Scale Language Modeling with Mixtures of Experts
This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning.
Few-shot Learning with Multilingual Language Models
Large-scale generative language models such as GPT-3 are competitive few-shot learners.
Sustainable AI: Environmental Implications, Challenges and Opportunities
This paper explores the environmental impact of the super-linear growth trends for AI from a holistic perspective, spanning Data, Algorithms, and System Hardware.
NormFormer: Improved Transformer Pretraining with Extra Normalization
The extra operations incur negligible compute cost (+0. 4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2. 7 Billion parameters.
On Anytime Learning at Macroscale
Practitioners have then to decide how to allocate their computational budget in order to obtain the best performance at any point in time.
Larger-Scale Transformers for Multilingual Masked Language Modeling
Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0. 3% on average while handling 99 more languages.
Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models
We find that mix-review effectively regularizes the finetuning process, and the forgetting problem is alleviated to some extent.
Few-shot Sequence Learning with Transformers
In the simplest setting, we append a token to an input sequence which represents the particular task to be undertaken, and show that the embedding of this token can be optimized on the fly given few labeled examples.
Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art
A large array of pretrained models are available to the biomedical NLP (BioNLP) community.
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation.
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
The resulting method offers a compelling solution for using large-scale pre-trained models at a fraction of the computational cost when multiple tasks are performed on the same text.
Recipes for building an open-domain chatbot
Building open-domain chatbots is a challenging area for machine learning research.
Residual Energy-Based Models for Text Generation
In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level.
Residual Energy-Based Models for Text
Current large-scale auto-regressive language models display impressive fluency and can generate convincing text.
How Decoding Strategies Affect the Verifiability of Generated Text
A generated sentence is verifiable if it can be corroborated or disproved by Wikipedia, and we find that the verifiability of generated text strongly depends on the decoding strategy.
Unsupervised Cross-lingual Representation Learning at Scale
We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale.
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali--English and Sinhala--English
For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little parallel data available.
Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
We find that mix-review effectively regularizes the finetuning process, and the forgetting problem is alleviated to some extent.
Facebook AI's WAT19 Myanmar-English Translation Task Submission
This paper describes Facebook AI's submission to the WAT 2019 Myanmar-English translation task.
The Source-Target Domain Mismatch Problem in Machine Translation
While we live in an increasingly interconnected world, different places still exhibit strikingly different cultures and many events we experience in our every day life pertain only to the specific place we live in.
On The Evaluation of Machine Translation Systems Trained With Back-Translation
Back-translation is a widely used data augmentation technique which leverages target monolingual data.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging.
Ranked #1 on
Only Connect Walls Dataset Task 1 (Grouping)
on OCW
(Wasserstein Distance (WD) metric, using extra
training data)
Facebook FAIR's WMT19 News Translation Task Submission
This paper describes Facebook FAIR's submission to the WMT19 shared news translation task.
Ranked #1 on
Machine Translation
on WMT2019 English-German
Real or Fake? Learning to Discriminate Machine from Human Generated Text
Energy-based models (EBMs), a. k. a.
Diverse Machine Translation with a Single Multinomial Latent Variable
There are many ways to translate a sentence into another language.
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks.
Mixture Models for Diverse Machine Translation: Tricks of the Trade
Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English
For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little parallel data available.
Phrase-Based \& Neural Unsupervised Machine Translation
Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs.
Understanding Back-Translation at Scale
An effective method to improve neural machine translation with monolingual data is to augment the parallel training corpus with back-translations of target language sentences.
Ranked #2 on
Machine Translation
on WMT2014 English-German
(using extra training data)
Scaling Neural Machine Translation
Sequence to sequence learning models still require several days to reach state of the art performance on large benchmark datasets using a single machine.
Ranked #12 on
Machine Translation
on WMT2014 English-French
Phrase-Based & Neural Unsupervised Machine Translation
Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs.
Ranked #2 on
Machine Translation
on WMT2016 English-Russian
Analyzing Uncertainty in Neural Machine Translation
We propose tools and metrics to assess how uncertainty in the data is captured by the model distribution and how it affects search strategies that generate translations.
Classical Structured Prediction Losses for Sequence to Sequence Learning
There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam.
Ranked #4 on
Machine Translation
on IWSLT2015 German-English
