Search Results for author:
Found 36 papers, 10 papers with code
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on XTREME
(AVG metric)
Findings of the Shared Task on Machine Translation in Dravidian languages
This paper describes the datasets used, the methodology used for the evaluation of participants, and the experiments’ overall results.
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
We evaluate commonly used models on the benchmark.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
The unreasonable effectiveness of few-shot learning for machine translation
We demonstrate the potential of few-shot translation systems, trained with unpaired language data, for both high and low-resource language pairs.
Mu$^{2}$SLAM: Multitask, Multilingual Speech and Language Models
We present Mu$^{2}$SLAM, a multilingual sequence-to-sequence model pre-trained jointly on unlabeled speech, unlabeled text and supervised data spanning Automatic Speech Recognition (ASR), Automatic Speech Translation (AST) and Machine Translation (MT), in over 100 languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
XTREME-S: Evaluating Cross-lingual Speech Representations
Covering 102 languages from 10+ language families, 3 different domains and 4 task families, XTREME-S aims to simplify multilingual speech representation evaluation, as well as catalyze research in "universal" speech representation learning.
mSLAM: Massively multilingual joint pre-training for speech and text
We present mSLAM, a multilingual Speech and LAnguage Model that learns cross-lingual cross-modal representations of speech and text by pre-training jointly on large amounts of unlabeled speech and text in multiple languages.
Ranked #1 on
Spoken language identification
on FLEURS
(using extra training data)
DOCmT5: Document-Level Pretraining of Multilingual Language Models
In this paper, we introduce DOCmT5, a multilingual sequence-to-sequence language model pretrained with large scale parallel documents.
Ranked #1 on
Document Translation
on IWSLT2015
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
We build a single encoder with the BERT objective on unlabeled text together with the w2v-BERT objective on unlabeled speech.
Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents
Using simple concatenation-based DocNMT, we explore the effect of 3 factors on the transfer: the number of teacher languages with document level data, the balance between document and sentence level data at training, and the data condition of parallel documents (genuine vs. backtranslated).
HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints
For such cases, we propose training the model with additional hints (as target tags on the decoder) that provide information about the operation required on the source (translation or both translation and transliteration).
nmT5 - Is parallel data still relevant for pre-training massively multilingual language models?
Recently, mT5 - a massively multilingual version of T5 - leveraged a unified text-to-text format to attain state-of-the-art results on a wide variety of multilingual NLP tasks.
MergeDistill: Merging Pre-trained Language Models using Distillation
Pre-trained multilingual language models (LMs) have achieved state-of-the-art results in cross-lingual transfer, but they often lead to an inequitable representation of languages due to limited capacity, skewed pre-training data, and sub-optimal vocabularies.
nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
Recently, mT5 - a massively multilingual version of T5 - leveraged a unified text-to-text format to attain state-of-the-art results on a wide variety of multilingual NLP tasks.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
While a sizeable gap to human-level performance remains, improvements have been easier to achieve in some tasks than in others.
Gradient-guided Loss Masking for Neural Machine Translation
To mitigate the negative effect of low quality training data on the performance of neural machine translation models, most existing strategies focus on filtering out harmful data before training starts.
They, Them, Theirs: Rewriting with Gender-Neutral English
Responsible development of technology involves applications being inclusive of the diverse set of users they hope to support.
Distilling Large Language Models into Tiny and Effective Students using pQRNN
Our strong results suggest that our approach is great for latency-sensitive applications while being able to leverage large mBERT-like models.
Rethinking embedding coupling in pre-trained language models
We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art pre-trained language models.
Ranked #1 on
Cross-Lingual NER
on NER
Cross-Lingual Natural Language Inference
Cross-Lingual NER
+4
Explicit Alignment Objectives for Multilingual Bidirectional Encoders
Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020) have proven to be impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resource languages.
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing.
Adaptive Scheduling for Multi-Task Learning
To train neural machine translation models simultaneously on multiple tasks (languages), it is common to sample each task uniformly or in proportion to dataset sizes.
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
The recently proposed massively multilingual neural machine translation (NMT) system has been shown to be capable of translating over 100 languages to and from English within a single model.
Small and Practical BERT Models for Sequence Labeling
We propose a practical scheme to train a single multilingual sequence labeling model that yields state of the art results and is small and fast enough to run on a single CPU.
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
We introduce our efforts towards building a universal neural machine translation (NMT) system capable of translating between any language pair.
Direct speech-to-speech translation with a sequence-to-sequence model
We present an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation.
The Missing Ingredient in Zero-Shot Neural Machine Translation
Multilingual Neural Machine Translation (NMT) models are capable of translating between multiple source and target languages.
Massively Multilingual Neural Machine Translation
Our experiments on a large-scale dataset with 102 languages to and from English and up to one million examples per direction also show promising results, surpassing strong bilingual baselines and encouraging future work on massively multilingual NMT.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation
In this paper, we demonstrate that using pre-trained MT or text-to-speech (TTS) synthesis models to convert weakly supervised data into speech-to-translation pairs for ST training can be more effective than multi-task learning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
However, little attention has been paid to leveraging representations learned by a multilingual NMT system to enable zero-shot multilinguality in other NLP tasks.
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
Each of these new approaches consists of a fundamental architecture accompanied by a set of modeling and training techniques that are in principle applicable to other seq2seq architectures.
Ranked #26 on
Machine Translation
on WMT2014 English-French
Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
In addition to improving the translation quality of language pairs that the model was trained with, our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation.
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder.
Ranked #35 on
Machine Translation
on WMT2014 English-French
