Translatotron 3: Speech to Speech Translation with Monolingual Data

no code implementations • 27 May 2023 • Eliya Nachmani, Alon Levkovitch, Yifan Ding, Chulayuth Asawaroengchai, Heiga Zen, Michelle Tadmor Ramanovich

This paper presents Translatotron 3, a novel approach to unsupervised direct speech-to-speech translation from monolingual speech-text datasets by combining masked autoencoder, unsupervised embedding mapping, and back-translation.

Speech-to-Speech Translation Translation

Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

no code implementations • 24 May 2023 • Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, Michelle Tadmor Ramanovich

Key to our approach is a training objective that jointly supervises speech recognition, text continuation, and speech synthesis using only paired speech-text pairs, enabling a `cross-modal' chain-of-thought within a single decoding pass.

Language Modelling Question Answering +3

CVSS Corpus and Massively Multilingual Speech-to-Speech Translation

1 code implementation • LREC 2022 • Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, Heiga Zen

In addition, CVSS provides normalized translation text which matches the pronunciation in the translation speech.

Sentence Speech-to-Speech Translation +2

More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech

no code implementations • CVPR 2022 • Michael Hassid, Michelle Tadmor Ramanovich, Brendan Shillingford, Miaosen Wang, Ye Jia, Tal Remez

In this paper we present VDTTS, a Visually-Driven Text-to-Speech model.

Translatotron 2: High-quality direct speech-to-speech translation with voice preservation

no code implementations • 19 Jul 2021 • Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, Roi Pomerantz

We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end.

Data Augmentation Speech-to-Speech Translation +2