Search Results for author:
Found 41 papers, 18 papers with code
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq Sˆ2, a fairseq extension for speech synthesis.
Data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The lip-reading WER is further reduced to 26. 9% when using all 433 hours of labeled data from LRS3 and combined with self-training.
Ranked #1 on
Lipreading
on LRS3-TED
(using extra training data)
Robust Self-Supervised Audio-Visual Speech Recognition
Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe.
Ranked #1 on
Audio-Visual Speech Recognition
on LRS3-TED
Textless Speech-to-Speech Translation on Real Data
To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs.
Textless Speech Emotion Conversion using Decomposed & Discrete Representations
We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion.
Textless Speech Emotion Conversion using Decomposed and Discrete Representations
We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion.
Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention
We present a direct simultaneous speech-to-speech translation (Simul-S2ST) model, Furthermore, the generation of translation is independent from intermediate text representations.
fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq S^2, a fairseq extension for speech synthesis.
Text-Free Prosody-Aware Generative Spoken Language Modeling
Generative Spoken Language Modeling (GSLM) (Lakhotia et al., 2021) is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences.
Direct speech-to-speech translation with discrete units
We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation.
Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition
In this paper, we introduce the Kaizen framework that uses a continuously improving teacher to generate pseudo-labels for semi-supervised speech recognition (ASR).
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation.
Unsupervised Speech Recognition
Despite rapid progress in the recent past, current speech recognition systems still require labeled training data which limits this technology to a small fraction of the languages spoken around the globe.
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
On a large-scale competitive setup, we show that pre-training on unlabeled in-domain data reduces the gap between models trained on in-domain and out-of-domain labeled data by 66%-73%.
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
We propose using self-supervised discrete representations for the task of speech resynthesis.
Generative Spoken Language Modeling from Raw Audio
We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation.
Ranked #1 on
Resynthesis
on LJSpeech
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units
In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision.
Differentiable Weighted Finite-State Transducers
We introduce a framework for automatic differentiation with weighted finite-state transducers (WFSTs) allowing them to be used dynamically at training time.
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning
Probabilistic Latent Variable Models (LVMs) provide an alternative to self-supervised learning approaches for linguistic representation learning from speech.
Semi-Supervised Speech Recognition via Local Prior Matching
For sequence transduction tasks like speech recognition, a strong structured prior model encodes rich information about the target space, implicitly ruling out invalid sequences by assigning them low probability.
Ranked #38 on
Speech Recognition
on LibriSpeech test-other
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
What differentiates this paper from prior work on speech unit learning is the choice of training objective.
Self-Supervised Speech Recognition via Local Prior Matching
We propose local prior matching (LPM), a self-supervised objective for speech recognition.
Transfer Learning from Audio-Visual Grounding to Speech Recognition
Transfer learning aims to reduce the amount of data required to excel at a new task by re-using the knowledge acquired from learning other related tasks.
An Unsupervised Autoregressive Model for Speech Representation Learning
This paper proposes a novel unsupervised autoregressive neural model for learning generic speech representations.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
Hierarchical Generative Modeling for Controllable Speech Synthesis
This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions.
Unsupervised Representation Learning of Speech for Dialect Identification
In this paper, we explore the use of a factorized hierarchical variational autoencoder (FHVAE) model to learn an unsupervised latent representation for dialect identification (DID).
Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis
We demonstrate that the proposed framework enables Tacotron to generate intelligible speech using less than half an hour of paired training data.
Unsupervised Adaptation with Interpretable Disentangled Representations for Distant Conversational Speech Recognition
However, it is relatively inexpensive to collect large amounts of unlabeled data from domains that we want the models to generalize to.
A Study of Enhancement, Augmentation, and Autoencoder Methods for Domain Adaptation in Distant Speech Recognition
Speech recognizers trained on close-talking speech do not generalize to distant speech and the word error rate degradation can be as large as 40% absolute.
Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data
In this paper, we present a partitioned variational autoencoder (PVAE) and several training objectives to learn disentangled representations, which encode not only the shared factors, but also modality-dependent ones, into separate latent variables.
Scalable Factorized Hierarchical Variational Autoencoder Training
Deep generative models have achieved great success in unsupervised learning with the ability to capture complex nonlinear relationships between latent generating factors and observations.
Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition
The performance of automatic speech recognition (ASR) systems can be significantly compromised by previously unseen conditions, which is typically due to a mismatch between training and testing distributions.
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
We present a factorized hierarchical variational autoencoder, which learns disentangled and interpretable representations from sequential data without supervision.
Unsupervised Domain Adaptation for Robust Speech Recognition via Variational Autoencoder-Based Data Augmentation
Research on robust speech recognition can be regarded as trying to overcome this domain mismatch issue.
Learning Latent Representations for Speech Generation and Transformation
In this paper, we apply a convolutional VAE to model the generative process of natural speech.
Neural Attention for Learning to Rank Questions in Community Question Answering
In real-world data, e. g., from Web forums, text is often contaminated with redundant or irrelevant content, which leads to introducing noise in machine learning algorithms.
Recurrent Neural Network Encoder with Attention for Community Question Answering
We apply a general recurrent neural network (RNN) encoder framework to community question answering (cQA) tasks.
Enhancing Automatically Discovered Multi-level Acoustic Patterns Considering Context Consistency With Applications in Spoken Term Detection
This paper presents a novel approach for enhancing the multiple sets of acoustic patterns automatically discovered from a given corpus.
