Search Results for author:
Found 42 papers, 22 papers with code
Findings of the WMT 2021 Shared Task on Large-Scale Multilingual Machine Translation
There were a total of 10 participating teams for the tasks, with a total of 151 intermediate model submissions and 13 final models.
Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment
Following two preceding WMT Shared Task on Parallel Corpus Filtering (Koehn et al., 2018, 2019), we posed again the challenge of assigning sentence-level quality scores for very noisy corpora of sentence pairs crawled from the web, with the goal of sub-selecting the highest-quality data to be used to train ma-chine translation systems.
Facebook AI’s WMT20 News Translation Task Submission
We approach the low resource problem using two main strategies, leveraging all available data and adapting the system to the target news domain.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Question Answering
on PeerQA
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
We use this dataset to evaluate the capabilities of multilingual masked language models (MLMs) and large language models (LLMs).
Llama 2: Open Foundation and Fine-Tuned Chat Models
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.
Ranked #2 on
Question Answering
on PubChemQA
A Theory on Adam Instability in Large-Scale Machine Learning
We present a theory for the previously unexplained divergent behavior noticed in the training of large language models.
LLaMA: Open and Efficient Foundation Language Models
We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters.
Ranked #3 on
Few-Shot Learning
on MedConceptsQA
Text-To-4D Dynamic Scene Generation
We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions.
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
Large multilingual language models typically rely on a single vocabulary shared across 100+ languages.
Scaling Laws for Generative Mixed-Modal Language Models
To better understand the scaling properties of such mixed-modal models, we conducted over 250 experiments using seven different modalities and model sizes ranging from 8 million to 30 billion, trained on 5-100 billion tokens.
A survey on Self Supervised learning approaches for improving Multimodal representation learning
Recently self supervised learning has seen explosive growth and use in variety of machine learning tasks because of its ability to avoid the cost of annotating large-scale datasets.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We present BlenderBot 3, a 175B parameter dialogue model capable of open-domain conversation with access to the internet and a long-term memory, and having been trained on a large number of user defined tasks.
On the Role of Bidirectionality in Language Model Pre-Training
Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult.
Lifting the Curse of Multilinguality by Pre-training Modular Transformers
Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages.
OPT: Open Pre-trained Transformer Language Models
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning.
Ranked #2 on
Stereotypical Bias Analysis
on CrowS-Pairs
How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?
Since a skewed data distribution is considered to be harmful, a sampling strategy is usually used to balance languages in the corpus.
Graph Neural Networks for Image Classification and Reinforcement Learning using Graph representations
In this paper, we will evaluate the performance of graph neural networks in two distinct domains: computer vision and reinforcement learning.
CM3: A Causal Masked Multimodal Model of the Internet
We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens.
Few-shot Learning with Multilingual Language Models
Large-scale generative language models such as GPT-3 are competitive few-shot learners.
Efficient Large Scale Language Modeling with Mixtures of Experts
This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning.
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7. 4 BLEU over 21 translation directions into English.
Ranked #1 on
Language Identification
on VOXLINGUA107
FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Task
In this paper, we describe our end-to-end multilingual speech translation system submitted to the IWSLT 2021 evaluation campaign on the Multilingual Speech Translation shared task.
The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks.
Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data
The scarcity of parallel data is a major obstacle for training high-quality machine translation systems for low-resource languages.
Larger-Scale Transformers for Multilingual Masked Language Modeling
Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0. 3% on average while handling 99 more languages.
BASE Layers: Simplifying Training of Large, Sparse Models
Sparse layers can dramatically improve the efficiency of training and inference by routing each token to specialized expert modules that contain only a small fraction of the model parameters.
Multilingual Autoregressive Entity Linking
Moreover, in a zero-shot setting on languages with no training data at all, mGENRE treats the target language as a latent variable that is marginalized at prediction time.
Ranked #2 on
Entity Disambiguation
on Mewsli-9
(using extra training data)
Facebook AI's WMT20 News Translation Task Submission
We approach the low resource problem using two main strategies, leveraging all available data and adapting the system to the target news domain.
Beyond English-Centric Multilingual Machine Translation
Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages.
Better Fine-Tuning by Reducing Representational Collapse
Although widely adopted, existing approaches for fine-tuning pre-trained language models have been shown to be unstable across hyper-parameter settings, motivating recent work on trust region methods.
Abstractive Text Summarization
Cross-Lingual Natural Language Inference
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
Recent work demonstrates the potential of multilingual pretraining of creating one model that can be used for various tasks in different languages.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks.
Ranked #6 on
Fact Verification
on FEVER
(using extra training data)
Recipes for building an open-domain chatbot
Building open-domain chatbots is a challenging area for machine learning research.
Unsupervised Cross-lingual Representation Learning at Scale
We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale.
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token.
Ranked #3 on
Open-Domain Question Answering
on ELI5
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging.
Ranked #1 on
Only Connect Walls Dataset Task 1 (Grouping)
on OCW
(Wasserstein Distance (WD) metric, using extra
training data)
LearningToQuestion at SemEval 2017 Task 3: Ranking Similar Questions by Learning to Rank Using Rich Features
This paper describes our official entry LearningToQuestion for SemEval 2017 task 3 community question answer, subtask B.
The Social Dynamics of Language Change in Online Networks
Language change is a complex social phenomenon, revealing pathways of communication and sociocultural influence.
