Search Results for author:
Found 73 papers, 30 papers with code
Enhancing Automatically Discovered Multi-level Acoustic Patterns Considering Context Consistency With Applications in Spoken Term Detection
This paper presents a novel approach for enhancing the multiple sets of acoustic patterns automatically discovered from a given corpus.
Recurrent Neural Network Encoder with Attention for Community Question Answering
We apply a general recurrent neural network (RNN) encoder framework to community question answering (cQA) tasks.
Neural Attention for Learning to Rank Questions in Community Question Answering
In real-world data, e. g., from Web forums, text is often contaminated with redundant or irrelevant content, which leads to introducing noise in machine learning algorithms.
Learning Latent Representations for Speech Generation and Transformation
In this paper, we apply a convolutional VAE to model the generative process of natural speech.
Unsupervised Domain Adaptation for Robust Speech Recognition via Variational Autoencoder-Based Data Augmentation
Research on robust speech recognition can be regarded as trying to overcome this domain mismatch issue.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
We present a factorized hierarchical variational autoencoder, which learns disentangled and interpretable representations from sequential data without supervision.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition
The performance of automatic speech recognition (ASR) systems can be significantly compromised by previously unseen conditions, which is typically due to a mismatch between training and testing distributions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Scalable Factorized Hierarchical Variational Autoencoder Training
Deep generative models have achieved great success in unsupervised learning with the ability to capture complex nonlinear relationships between latent generating factors and observations.
Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data
In this paper, we present a partitioned variational autoencoder (PVAE) and several training objectives to learn disentangled representations, which encode not only the shared factors, but also modality-dependent ones, into separate latent variables.
Unsupervised Adaptation with Interpretable Disentangled Representations for Distant Conversational Speech Recognition
However, it is relatively inexpensive to collect large amounts of unlabeled data from domains that we want the models to generalize to.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A Study of Enhancement, Augmentation, and Autoencoder Methods for Domain Adaptation in Distant Speech Recognition
Speech recognizers trained on close-talking speech do not generalize to distant speech and the word error rate degradation can be as large as 40% absolute.
Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis
We demonstrate that the proposed framework enables Tacotron to generate intelligible speech using less than half an hour of paired training data.
Unsupervised Representation Learning of Speech for Dialect Identification
In this paper, we explore the use of a factorized hierarchical variational autoencoder (FHVAE) model to learn an unsupervised latent representation for dialect identification (DID).
Hierarchical Generative Modeling for Controllable Speech Synthesis
This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
An Unsupervised Autoregressive Model for Speech Representation Learning
This paper proposes a novel unsupervised autoregressive neural model for learning generic speech representations.
Transfer Learning from Audio-Visual Grounding to Speech Recognition
Transfer learning aims to reduce the amount of data required to excel at a new task by re-using the knowledge acquired from learning other related tasks.
Self-Supervised Speech Recognition via Local Prior Matching
We propose local prior matching (LPM), a self-supervised objective for speech recognition.
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
What differentiates this paper from prior work on speech unit learning is the choice of training objective.
Semi-Supervised Speech Recognition via Local Prior Matching
For sequence transduction tasks like speech recognition, a strong structured prior model encodes rich information about the target space, implicitly ruling out invalid sequences by assigning them low probability.
Ranked #45 on
Speech Recognition
on LibriSpeech test-other
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning
Probabilistic Latent Variable Models (LVMs) provide an alternative to self-supervised learning approaches for linguistic representation learning from speech.
Differentiable Weighted Finite-State Transducers
We introduce a framework for automatic differentiation with weighted finite-state transducers (WFSTs) allowing them to be used dynamically at training time.
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units
In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision.
Generative Spoken Language Modeling from Raw Audio
We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation.
Ranked #1 on
Resynthesis
on LibriSpeech
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
We propose using self-supervised discrete representations for the task of speech resynthesis.
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
On a large-scale competitive setup, we show that pre-training on unlabeled in-domain data reduces the gap between models trained on in-domain and out-of-domain labeled data by 66%-73%.
Unsupervised Speech Recognition
Despite rapid progress in the recent past, current speech recognition systems still require labeled training data which limits this technology to a small fraction of the languages spoken around the globe.
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation.
Ranked #4 on
Speech Recognition
on LibriSpeech test-other
Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition
In this paper, we introduce the Kaizen framework that uses a continuously improving teacher to generate pseudo-labels for semi-supervised speech recognition (ASR).
Direct speech-to-speech translation with discrete units
When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass.
Text-Free Prosody-Aware Generative Spoken Language Modeling
Generative Spoken Language Modeling (GSLM) \cite{Lakhotia2021} is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences.
fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq S^2, a fairseq extension for speech synthesis.
Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention
We present a direct simultaneous speech-to-speech translation (Simul-S2ST) model, Furthermore, the generation of translation is independent from intermediate text representations.
Textless Speech Emotion Conversion using Discrete and Decomposed Representations
We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion.
Textless Speech Emotion Conversion using Decomposed & Discrete Representations
We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion.
Textless Speech-to-Speech Translation on Real Data
To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The lip-reading WER is further reduced to 26. 9% when using all 433 hours of labeled data from LRS3 and combined with self-training.
Ranked #1 on
Speech Recognition
on LRS3-TED
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Robust Self-Supervised Audio-Visual Speech Recognition
Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe.
Ranked #2 on
Audio-Visual Speech Recognition
on LRS3-TED
(using extra training data)
Audio-Visual Speech Recognition
Automatic Speech Recognition
+5
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind.
Ranked #1 on
Paraphrase Identification
on Quora Question Pairs
(Accuracy metric)
textless-lib: a Library for Textless Spoken Language Processing
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources.
Measuring the Impact of Individual Domain Factors in Self-Supervised Pre-Training
In this paper, we present a controlled study to better understand the effect of such factors on the performance of pre-trained representations on automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Generative Spoken Dialogue Language Modeling
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues.
Towards End-to-end Unsupervised Speech Recognition
Unsupervised speech recognition has shown great potential to make Automatic Speech Recognition (ASR) systems accessible to every language.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Simple and Effective Unsupervised Speech Synthesis
We introduce the first unsupervised speech synthesis system based on a simple, yet effective recipe.
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation
Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues as there exists little parallel S2ST data, compared to the amount of data available for conventional cascaded systems that consist of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) synthesis.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Unified Speech-Text Pre-training for Speech Translation and Recognition
Two pre-training configurations for speech translation and recognition, respectively, are presented to alleviate subtask interference.
On-demand compute reduction with stochastic wav2vec 2.0
Our results for models pre-trained on 960h Librispeech dataset and fine-tuned on 10h of transcribed data show that using the same stochastic model, we get a smooth trade-off between word error rate (WER) and inference time with only marginal WER degradation compared to the W2V2 and SEW models trained for a specific setting.
Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT
This paper investigates self-supervised pre-training for audio-visual speaker representation learning where a visual stream showing the speaker's mouth area is used alongside speech as inputs.
STOP: A dataset for Spoken Task Oriented Semantic Parsing
Furthermore, in addition to the human-recorded audio, we are releasing a TTS-generated version to benchmark the performance for low-resource domain adaptation of end-to-end SLU systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality
By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-art modality-specific models.
Simple and Effective Unsupervised Speech Translation
The amount of labeled data to train models for speech tasks is limited for most languages, however, the data scarcity is exacerbated for speech translation which requires labeled data covering two different languages.
Speech-to-Speech Translation For A Real-world Unwritten Language
We use English-Taiwanese Hokkien as a case study, and present an end-to-end solution from training data collection, modeling choices to benchmark dataset release.
Continual Learning for On-Device Speech Recognition using Disentangled Conformers
Additionally, current speech recognition models and continual learning algorithms are not optimized to be compute-efficient.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language
Current self-supervised learning algorithms are often modality-specific and require large amounts of computational resources.
Ranked #91 on
Image Classification
on ImageNet
Efficient Speech Representation Learning with Low-Bit Quantization
With the development of hardware for machine learning, newer models often come at the cost of both increased sizes and computational complexity.
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR.
Ranked #1 on
Speech Recognition
on EasyCom
ReVISE: Self-Supervised Speech Resynthesis With Visual Input for Universal and Generalized Speech Regeneration
Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR.
Scaling Laws for Generative Mixed-Modal Language Models
To better understand the scaling properties of such mixed-modal models, we conducted over 250 experiments using seven different modalities and model sizes ranging from 8 million to 30 billion, trained on 5-100 billion tokens.
AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations
Self-supervision has shown great potential for audio-visual speech recognition by vastly reducing the amount of labeled data required to build good systems.
MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages.
Audio-Visual Speech Recognition
Robust Speech Recognition
+4
Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speech
Self-supervised learning leverages unlabeled data effectively, improving label efficiency and generalization to domains without labeled data.
DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning
In this paper, we introduce self-distillation and online clustering for self-supervised speech representation learning (DinoSR) which combines masked language modeling, self-distillation, and online clustering.
Scaling Speech Technology to 1,000+ Languages
Expanding the language coverage of speech technology has the potential to improve access to information for many more people.
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization).
Low-Resource Self-Supervised Learning with SSL-Enhanced TTS
Self-supervised learning (SSL) techniques have achieved remarkable results in various speech processing tasks.
Toward Joint Language Modeling for Speech Units and Text
However, in the field of language modeling, very little effort has been made to model them jointly.
Generative Pre-training for Speech with Flow Matching
Generative models have gained more and more attention in recent years for their remarkable success in tasks that required estimating and sampling data distribution to generate high-fidelity synthetic data.
Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiency
In this paper, we show that a simple self-supervised pre-trained audio model can achieve comparable inference efficiency to more complicated pre-trained models with speech transformer encoders.
Audiobox: Unified Audio Generation with Natural Language Prompts
Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data.
Ranked #1 on
Audio Generation
on AudioCaps
XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception
It is designed to maximize the benefits of limited multilingual AV pre-training data, by building on top of audio-only multilingual pre-training and simplifying existing pre-training schemes.
Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization
These models do not explicitly focus on the presence of concepts or events and their temporal ordering in the output audio with respect to the input prompt.
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq Sˆ2, a fairseq extension for speech synthesis.
