Search Results for author:
Found 18 papers, 8 papers with code
Facebook AI’s WMT21 News Translation Task Submission
We describe Facebook’s multilingual model submission to the WMT2021 shared task on news translation.
Effective Long-Context Scaling of Foundation Models
We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.
Llama 2: Open Foundation and Fine-Tuned Chat Models
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.
Ranked #1 on
on
Revisiting Machine Translation for Cross-lingual Classification
Machine Translation (MT) has been widely used for cross-lingual classification, either by translating the test set into English and running inference with a monolingual model (translate-test), or translating the training set into the target languages and finetuning a multilingual model (translate-train).
Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
We propose three optimization techniques to mitigate sources of inefficiencies, namely (1) Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing.
Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation
Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation.
Causes and Cures for Interference in Multilingual Translation
Multilingual machine translation models can benefit from synergy between different language pairs, but also suffer from interference.
No Language Left Behind: Scaling Human-Centered Machine Translation
Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.
Ranked #1 on
Machine Translation
on IWSLT2015 English-Vietnamese
(SacreBLEU metric)
Multilingual Machine Translation with Hyper-Adapters
We find that hyper-adapters are more parameter efficient than regular adapters, reaching the same performance with up to 12 times less parameters.
Data Selection Curriculum for Neural Machine Translation
In this work, we introduce a two-stage curriculum training framework for NMT where we fine-tune a base NMT model on subsets of data, selected by both deterministic scoring using pre-trained methods and online scoring that considers prediction scores of the emerging NMT model.
Few-shot Learning with Multilingual Language Models
Large-scale generative language models such as GPT-3 are competitive few-shot learners.
Efficient Large Scale Language Modeling with Mixtures of Experts
This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning.
Tricks for Training Sparse Translation Models
Multi-task learning with an unbalanced data distribution skews model learning towards high resource tasks, especially when model capacity is fixed and fully shared across all tasks.
Facebook AI WMT21 News Translation Task Submission
We describe Facebook's multilingual model submission to the WMT2021 shared task on news translation.
BASE Layers: Simplifying Training of Large, Sparse Models
Sparse layers can dramatically improve the efficiency of training and inference by routing each token to specialized expert modules that contain only a small fraction of the model parameters.
Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling
Pre-training models on vast quantities of unlabeled data has emerged as an effective approach to improving accuracy on many NLP tasks.
Ranked #1 on
Machine Translation
on WMT2016 Romanian-English
(using extra training data)
Beyond English-Centric Multilingual Machine Translation
Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages.
