Search Results for author:
Found 53 papers, 20 papers with code
Facebook AI’s WMT20 News Translation Task Submission
We approach the low resource problem using two main strategies, leveraging all available data and adapting the system to the target news domain.
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq Sˆ2, a fairseq extension for speech synthesis.
Findings of the IWSLT 2022 Evaluation Campaign
The evaluation campaign of the 19th International Conference on Spoken Language Translation featured eight shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Speech to speech translation, (iv) Low-resource speech translation, (v) Multilingual speech translation, (vi) Dialect speech translation, (vii) Formality control for speech translation, (viii) Isometric speech translation.
FINDINGS OF THE IWSLT 2021 EVALUATION CAMPAIGN
The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2021) featured this year four shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Multilingual speech translation, (iv) Low-resource speech translation.
DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models
Spoken language models (SLMs) have gained increasing attention with advancements in text-based, decoder-only language models.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Question Answering
on PeerQA
XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception
It is designed to maximize the benefits of limited multilingual AV pre-training data, by building on top of audio-only multilingual pre-training and simplifying existing pre-training schemes.
An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis
Speech language models (LMs) are promising for high-quality speech synthesis through in-context learning.
MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation
There have been emerging research interest and advances in speech-to-speech translation (S2ST), translating utterances from one language to another.
Seamless: Multilingual Expressive and Streaming Speech Translation
In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion.
automatic-speech-translation
Multimodal Machine Translation
+2
Exploring Speech Enhancement for Low-resource Speech Synthesis
High-quality and intelligible speech is essential to text-to-speech (TTS) model training, however, obtaining high-quality data for low-resource languages is challenging and expensive.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages?
Ranked #1 on
Speech-to-Speech Translation
on CVSS
(using extra training data)
Automatic Speech Recognition
Speech-to-Speech Translation
+4
Enhancing Speech-to-Speech Translation with Multiple TTS Targets
It has been known that direct speech-to-speech translation (S2ST) models usually suffer from the data scarcity issue because of the limited existing parallel materials for both source and target speech.
MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages.
Audio-Visual Speech Recognition
Robust Speech Recognition
+4
Pre-training for Speech Translation: CTC Meets Optimal Transport
Nevertheless, CTC is only a partial solution and thus, in our second contribution, we propose a novel pre-training method combining CTC and optimal transport to further reduce this gap.
A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation
Expressive speech-to-speech translation (S2ST) aims to transfer prosodic attributes of source speech to target speech while maintaining translation accuracy.
Introducing Semantics into Speech Encoders
Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+11
Speech-to-Speech Translation For A Real-world Unwritten Language
We use English-Taiwanese Hokkien as a case study, and present an end-to-end solution from training data collection, modeling choices to benchmark dataset release.
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations
We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings.
Improving Speech-to-Speech Translation Through Unlabeled Text
Direct speech-to-speech translation (S2ST) is among the most challenging problems in the translation paradigm due to the significant scarcity of S2ST data.
Simple and Effective Unsupervised Speech Translation
The amount of labeled data to train models for speech tasks is limited for most languages, however, the data scarcity is exacerbated for speech translation which requires labeled data covering two different languages.
Unified Speech-Text Pre-training for Speech Translation and Recognition
Two pre-training configurations for speech translation and recognition, respectively, are presented to alleviate subtask interference.
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation
Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues as there exists little parallel S2ST data, compared to the amount of data available for conventional cascaded systems that consist of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) synthesis.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+8
Textless Speech-to-Speech Translation on Real Data
To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs.
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7. 4 BLEU over 21 translation directions into English.
Ranked #1 on
Language Identification
on VOXLINGUA107
From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation
Speech-to-speech translation (S2ST) converts input speech to speech in another language.
Data Augmentation
Simultaneous Speech-to-Speech Translation
+4
Contrastive Clustering to Mine Pseudo Parallel Data for Unsupervised Translation
Modern unsupervised machine translation systems mostly train their models by generating synthetic parallel training data from large unlabeled monolingual corpora of different languages through various means, such as iterative back-translation.
fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq S^2, a fairseq extension for speech synthesis.
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models
We present a simple yet effective approach to build multilingual speech-to-text (ST) translation through efficient transfer learning from a pretrained speech encoder and text decoder.
FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Task
In this paper, we describe our end-to-end multilingual speech translation system submitted to the IWSLT 2021 evaluation campaign on the Multilingual Speech Translation shared task.
Direct speech-to-speech translation with discrete units
When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass.
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task
Pretraining and multitask learning are widely used to improve the speech to text translation performance.
Lightweight Adapter Tuning for Multilingual Speech Translation
Adapter modules were recently introduced as an efficient alternative to fine-tuning in NLP.
Ranked #1 on
Speech-to-Text Translation
on MuST-C EN->ES
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Large-Scale Self- and Semi-Supervised Learning for Speech Translation
In this paper, we improve speech translation (ST) through effectively leveraging large quantities of unlabeled speech and text data in different and complementary ways.
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
We introduce VoxPopuli, a large-scale multilingual corpus providing 100K hours of unlabelled speech data in 23 languages.
Ranked #3 on
Speech Recognition
on Common Voice French
(using extra training data)
Facebook AI's WMT20 News Translation Task Submission
We approach the low resource problem using two main strategies, leveraging all available data and adapting the system to the target news domain.
Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation
We propose two variants of these architectures corresponding to two different levels of dependencies between the decoders, called the parallel and cross dual-decoder Transformers, respectively.
Ranked #1 on
Speech-to-Text Translation
on MuST-C EN->FR
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
We present a simple yet effective approach to build multilingual speech-to-text (ST) translation by efficient transfer learning from pretrained speech encoder and text decoder.
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks
We demonstrate that representing text input as phoneme sequences can reduce the difference between speech and text inputs, and enhance the knowledge transfer from text corpora to the speech to text tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
fairseq S2T: Fast Speech-to-Text Modeling with fairseq
We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation.
Ranked #8 on
Speech-to-Text Translation
on MuST-C EN->DE
SimulEval: An Evaluation Toolkit for Simultaneous Translation
We also adapt latency metrics from text simultaneous translation to the speech task.
CoVoST 2 and Massively Multilingual Speech-to-Text Translation
Speech translation has recently become an increasingly popular topic of research, partly due to the development of benchmark datasets.
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN
The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2020) featured this year six challenge tracks: (i) Simultaneous speech translation, (ii) Video speech translation, (iii) Offline speech translation, (iv) Conversational speech translation, (v) Open domain translation, and (vi) Non-native speech translation.
Self-Supervised Representations Improve End-to-End Speech Translation
End-to-end speech-to-text translation can provide a simpler and smaller system but is facing the challenge of data scarcity.
Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation
Even with pseudo-labels from low-resource MT (200K examples), ST-enhanced transfer brings up to 8. 9% WER reduction to direct transfer.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+9
CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus
Spoken language translation has recently witnessed a resurgence in popularity, thanks to the development of end-to-end models and the creation of new corpora, such as Augmented LibriSpeech and MuST-C.
VizSeq: A Visual Analysis Toolkit for Text Generation Tasks
Automatic evaluation of text generation tasks (e. g. machine translation, text summarization, image captioning and video description) usually relies heavily on task-specific metrics, such as BLEU and ROUGE.
Neural Machine Translation with Byte-Level Subwords
Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue.
Does Object Recognition Work for Everyone?
The paper analyzes the accuracy of publicly available object-recognition systems on a geographically diverse dataset.
Levenshtein Transformer
We further confirm the flexibility of our model by showing a Levenshtein Transformer trained by machine translation can straightforwardly be used for automatic post-editing.
Ranked #7 on
Machine Translation
on WMT2016 Romanian-English
Code-Switched Named Entity Recognition with Embedding Attention
We describe our work for the CALCS 2018 shared task on named entity recognition on code-switched data.
Dynamic Meta-Embeddings for Improved Sentence Representations
While one of the first steps in many NLP systems is selecting what pre-trained word embeddings to use, we argue that such a step is better left for neural networks to figure out by themselves.
Ranked #52 on
Natural Language Inference
on SNLI
