Search Results for author:
Found 142 papers, 34 papers with code
Addressing speaker gender bias in large scale speech translation systems
This study addresses the issue of speaker gender bias in Speech Translation (ST) systems, which can lead to offensive and inaccurate translations.
SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training
Recent advancements highlight the potential of end-to-end real-time spoken dialogue systems, showcasing their low latency and high quality.
Interleaved Speech-Text Language Models are Simple Streaming Text to Speech Synthesizers
This paper introduces Interleaved Speech-Text Language Model (IST-LM) for streaming zero-shot Text-to-Speech (TTS).
AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM
The AlignFormer can achieve a near 100% IFR with audio-first training and game-changing improvements from zero to non-zero IFR on some evaluation data with instruction-first training.
V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow
In this paper, we introduce V2SFlow, a novel Video-to-Speech (V2S) framework designed to generate natural and intelligible speech directly from silent talking face videos.
TS3-Codec: Transformer-Based Simple Streaming Single Codec
Neural audio codecs (NACs) have garnered significant attention as key technologies for audio compression as well as audio representation for speech language models.
ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation
Text-to-video models have recently undergone rapid and substantial advancements.
Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation
To address these issues, we propose DARAG (Data- and Retrieval-Augmented Generative Error Correction), a novel approach designed to improve GEC for ASR in in-domain (ID) and OOD scenarios.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation
Models for streaming speech translation (ST) can achieve high accuracy and low latency if they're developed with vast amounts of paired audio in the source language and written text in the target language.
Target word activity detector: An approach to obtain ASR word boundaries without lexicon
In this work, we propose a new approach to estimate word boundaries without relying on lexicons.
Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech
This paper introduces EmoCtrl-TTS, an emotion-controllable zero-shot TTS that can generate highly emotional speech with NVs for any speaker.
Autoregressive Speech Synthesis without Vector Quantization
We present MELLE, a novel continuous-valued tokens based language modeling approach for text to speech synthesis (TTS).
Soft Language Identification for Language-Agnostic Many-to-One End-to-End Speech Translation
Language-agnostic many-to-one end-to-end speech translation models can convert audio signals from different source languages into text in a target language.
VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
With the help of discrete neural audio codecs, large language models (LLM) have increasingly been recognized as a promising methodology for zero-shot Text-to-Speech (TTS) synthesis.
An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTS
Recently, zero-shot text-to-speech (TTS) systems, capable of synthesizing any speaker's voice from a short audio prompt, have made rapid advancements.
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
This paper introduces VALL-E 2, the latest advancement in neural codec language models that marks a milestone in zero-shot text-to-speech synthesis (TTS), achieving human parity for the first time.
Total-Duration-Aware Duration Modeling for Text-to-Speech Systems
We also show that the proposed MaskGIT-based model can generate phoneme durations with higher quality and diversity compared to its regression or flow-matching counterparts.
TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation
There is a rising interest and trend in research towards directly translating speech from one language to another, known as end-to-end speech-to-speech translation.
CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
In this paper, we introduce CoVoMix: Conversational Voice Mixture Generation, a novel model for zero-shot, human-like, multi-speaker, multi-round dialogue speech generation.
RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis
Furthermore, we demonstrate that RALL-E correctly synthesizes sentences that are hard for VALL-E and reduces the error rate from $68\%$ to $4\%$.
WavLLM: Towards Robust and Adaptive Speech Large Language Model
In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach.
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
Specifically, 1) we design a neural codec with factorized vector quantization (FVQ) to disentangle speech waveform into subspaces of content, prosody, timbre, and acoustic details; 2) we propose a factorized diffusion model to generate attributes in each subspace following its corresponding prompt.
Boosting Large Language Model for Speech Synthesis: An Empirical Study
In this paper, we conduct a comprehensive empirical exploration of boosting LLMs with the ability to generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech synthesis model VALL-E. We compare three integration methods between LLMs and speech synthesis models, including directly fine-tuned LLMs, superposed layers of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text encoder.
COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning
We present a cost-effective method to integrate speech into a large language model (LLM), resulting in a Contextual Speech Model with Instruction-following/in-context-learning Capabilities (COSMIC) multi-modal LLM.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+8
RD-VIO: Robust Visual-Inertial Odometry for Mobile Augmented Reality in Dynamic Environments
In this work, we design a novel visual-inertial odometry (VIO) system called RD-VIO to handle both of these two problems.
Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translation
The growing need for instant spoken language transcription and translation is driven by increased global communication and cross-lingual interactions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Enhanced Edge-Perceptual Guided Image Filtering
Due to the powerful edge-preserving ability and low computational complexity, Guided image filter (GIF) and its improved versions has been widely applied in computer vision and image processing.
Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approach
Simultaneous Speech-to-Text translation serves a critical role in real-time crosslingual communication.
ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers
In this paper, we aim to reduce model size by reparameterizing model weights across Transformer encoder layers and assuming a special weight composition and structure.
t-SOT FNT: Streaming Multi-talker ASR with Text-only Domain Adaptation Capability
Token-level serialized output training (t-SOT) was recently proposed to address the challenge of streaming multi-talker automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
DiariST: Streaming Speech Translation with Speaker Diarization
End-to-end speech translation (ST) for conversation recordings involves several under-explored challenges such as speaker diarization (SD) without accurate word time stamps and handling of overlapping speech in a streaming fashion.
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Recent advancements in generative speech models based on audio-text prompts have enabled remarkable innovations like high-quality zero-shot text-to-speech.
Pre-training End-to-end ASR Models with Augmented Speech Samples Queried by Text
When mixing 20, 000 hours augmented speech data generated by our method with 12, 500 hours original transcribed speech data for Italian Transformer transducer model pre-training, we achieve 8. 7% relative word error rate reduction.
On decoder-only architecture for speech-to-text and large language model integration
Large language models (LLMs) have achieved remarkable success in the field of natural language processing, enabling better human-computer interaction using natural language.
Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments
In real-world applications, users often require both translations and transcriptions of speech to enhance their comprehension, particularly in streaming scenarios where incremental generation is necessary.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition
Different from these methods, in this work, with only a domain-specific text prompt, we propose two zero-shot ASR domain adaptation methods using LLaMA, a 7-billion-parameter large language model (LLM).
Accelerating Transducers through Adjacent Token Merging
Recent end-to-end automatic speech recognition (ASR) systems often utilize a Transformer-based acoustic encoder that generates embedding at a high frame rate.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Accurate and Structured Pruning for Efficient Automatic Speech Recognition
In this paper, we propose a novel compression strategy that leverages structured pruning and knowledge distillation to reduce the model size and inference cost of the Conformer model while preserving high recognition performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation
Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities.
PillarNeXt: Rethinking Network Designs for 3D Object Detection in LiDAR Point Clouds
In order to deal with the sparse and unstructured raw point clouds, LiDAR based 3D object detection research mostly focuses on designing dedicated local point aggregators for fine-grained geometrical modeling.
Ranked #1 on
3D Object Detection
on waymo vehicle
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis.
Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training
We propose gated language experts and curriculum training to enhance multilingual transformer transducer models without requiring language identification (LID) input from users during inference.
Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation
To solve above limitations, in this paper we propose an improved non-autoregressive (NAR) spelling correction model for contextual biasing in E2E neural transducer-based ASR systems to improve the previous CSC model from two perspectives: Firstly, we incorporate acoustics information with an external attention as well as text hypotheses into CSC to better distinguish target phrase from dissimilar or irrelevant phrases.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Speaker Change Detection for Transformer Transducer ASR
Speaker change detection (SCD) is an important feature that improves the readability of the recognized words from an automatic speech recognition (ASR) system by breaking the word sequence into paragraphs at speaker change points.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis.
Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models
Neural transducer is now the most popular end-to-end model for speech recognition, due to its naturally streaming ability.
VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
Although speech is a simple and effective way for humans to communicate with the outside world, a more realistic speech interaction contains multimodal information, e. g., vision, text.
LongFNT: Long-form Speech Recognition with Factorized Neural Transducer
This motivates us to leverage the factorized neural transducer structure, containing a real language model, the vocabulary predictor.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Self-supervised learning with bi-label masked speech prediction for streaming multi-talker speech recognition
In this paper, we investigate SSL for streaming multi-talker speech recognition, which generates transcriptions of overlapping speakers in a streaming fashion.
Speech separation with large-scale self-supervised learning
Compared with a supervised baseline and the WavLM-based SS model using feature embeddings obtained with the previously released 94K hours trained WavLM, our proposed model obtains 15. 9% and 11. 2% of relative word error rate (WER) reductions, respectively, for a simulated far-field speech mixture test set.
Streaming, fast and accurate on-device Inverse Text Normalization for Automatic Speech Recognition
Automatic Speech Recognition (ASR) systems typically yield output in lexical form.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers
In this paper, we propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability
In this paper, we introduce our work of building a Streaming Multilingual Speech Model (SM2), which can transcribe or translate multiple spoken languages into texts of the target language.
Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation
However, direct S2ST suffers from the data scarcity problem because the corpora from speech of the source language to speech of the target language are very rare.
Simulating realistic speech overlaps improves multi-talker ASR
Multi-talker automatic speech recognition (ASR) has been studied to generate transcriptions of natural conversation including overlapping speech of multiple speakers.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Acoustic-aware Non-autoregressive Spell Correction with Mask Sample Decoding
As a result, we reduce the WER of a streaming TT from 7. 6% to 6. 5% on the Librispeech test-other data and the CER from 7. 3% to 6. 1% on the Aishell test data, respectively.
CTCBERT: Advancing Hidden-unit BERT with CTC Objectives
We examine CTCBERT on IDs from HuBERT Iter1, HuBERT Iter2, and PBERT.
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
The rapid development of single-modal pre-training has prompted researchers to pay more attention to cross-modal pre-training methods.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data
In this paper, we propose a cross-modal Speech and Language Model (SpeechLM) to explicitly align speech and text pre-training with a pre-defined unified discrete representation.
VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition
To combine the best of both technologies, we newly design a t-SOT-based ASR model that generates a serialized multi-talker transcription based on two separated speech signals from VarArray.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Tower Bridge Net (TB-Net): Bidirectional Knowledge Graph Aware Embedding Propagation for Explainable Recommender Systems
Recently, neural networks based models have been widely used for recommender systems (RS).
D$^3$FlowSLAM: Self-Supervised Dynamic SLAM with Flow Motion Decomposition and DINO Guidance
We propose a dynamic update module based on this representation and develop a dense SLAM system that excels in dynamic scenarios.
Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training
Recently, masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
The YiTrans End-to-End Speech Translation System for IWSLT 2022 Offline Shared Task
The YiTrans system is built on large-scale pre-trained encoder-decoder models.
Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?
Recently, self-supervised learning (SSL) has demonstrated strong performance in speaker recognition, even if the pre-training objective is designed for speech recognition.
Ultra Fast Speech Separation Model with Teacher Student Learning
In this paper, an ultra fast speech separation Transformer model is proposed to achieve both better performance and efficiency with teacher student learning (T-S learning).
Large-Scale Streaming End-to-End Speech Translation with Neural Transducers
Neural transducers have been widely used in automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data
In this way, the decoder learns to reconstruct original speech information with codes before learning to generate correct text.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings
The proposed speaker embedding, named t-vector, is extracted synchronously with the t-SOT ASR model, enabling joint execution of speaker identification (SID) or speaker diarization (SD) with the multi-talker transcription with low latency.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Towards Contextual Spelling Correction for Customization of End-to-end Speech Recognition Systems
In this work, we introduce a novel approach to do contextual biasing by adding a contextual spelling correction model on top of the end-to-end ASR system.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Streaming Multi-Talker ASR with Token-Level Serialized Output Training
This paper proposes a token-level serialized output training (t-SOT), a novel framework for streaming multi-talker automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Endpoint Detection for Streaming End-to-End Multi-talker ASR
Our experimental results based on the 2-speaker LibrispeechMix dataset show that the SURT model can achieve promising EP detection without significantly degradation of the recognition accuracy.
Self-Supervised Learning for speech recognition with Intermediate layer supervision
Recently, pioneer work finds that speech pre-trained models can solve full-stack speech processing tasks, because the model utilizes bottom layers to learn speaker-related information and top layers to encode content-related information.
Sequence-level self-learning with multiple hypotheses
For untranscribed speech data, the hypothesis from an ASR system must be used as a label.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Recent Advances in End-to-End Automatic Speech Recognition
Recently, the speech community is seeing a significant trend of moving from deep neural network based hybrid modeling to end-to-end (E2E) modeling for automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction
The reconstruction module is used for auxiliary learning to improve the noise robustness of the learned representation and thus is not required during inference.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+9
Continuous Speech Separation with Recurrent Selective Attention Network
In this paper, we propose to apply recurrent selective attention network (RSAN) to CSS, which generates a variable number of output channels based on active speaker counting.
Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Multi-talker conversational speech processing has drawn many interests for various applications such as meeting transcription.
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks.
Ranked #1 on
Speech Recognition
on CALLHOME En
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+8
UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
We integrate the proposed methods into the HuBERT framework.
Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition
In this paper we propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech via contrastive learning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+7
Have best of both worlds: two-pass hybrid and E2E cascading framework for speech recognition
Hybrid and end-to-end (E2E) systems have their individual advantages, with different error patterns in the speech recognition results.
Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition
ILMA enables a fast text-only adaptation of the E2E model without increasing the run-time computational cost.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Factorized Neural Transducer for Efficient Language Model Adaptation
In recent years, end-to-end (E2E) based automatic speech recognition (ASR) systems have achieved great success due to their simplicity and promising performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Continuous Streaming Multi-Talker ASR with Dual-path Transducers
Streaming recognition of multi-talker conversations has so far been evaluated only for 2-speaker single-turn sessions.
A Light-weight contextual spelling correction model for customizing transducer-based speech recognition systems
We incorporate the context information into the spelling correction model with a shared context encoder and use a filtering algorithm to handle large-size context lists.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A Configurable Multilingual Model is All You Need to Recognize All Languages
Particularly, a single CMM can be deployed to any user scenario where the users can pre-select any combination of languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
Recently, there has been a vast interest in self-supervised learning (SSL) where the model is pre-trained on large scale unlabeled data and then fine-tuned on a small labeled dataset.
Investigation of Practical Aspects of Single Channel Speech Separation for ASR
Speech separation has been successfully applied as a frontend processing module of conversation transcription systems thanks to its ability to handle overlapped speech and its flexibility to combine with downstream tasks such as automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition
In this work, we perform LM fusion in the minimum WER (MWER) training of an E2E model to obviate the need for LM weights tuning during inference.
On Addressing Practical Challenges for RNN-Transducer
The first challenge is solved with a splicing data method which concatenates the speech segments extracted from the source domain data.
Streaming Multi-talker Speech Recognition with Joint Speaker Identification
In multi-talker scenarios such as meetings and conversations, speech processing systems are usually required to transcribe the audio as well as identify the speakers for downstream applications.
Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition
The efficacy of external language model (LM) integration with existing end-to-end (E2E) automatic speech recognition (ASR) systems can be improved significantly using the internal language model estimation (ILME) method.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Streaming end-to-end multi-talker speech recognition
End-to-end multi-talker speech recognition is an emerging research trend in the speech community due to its vast potential in applications such as conversation and meeting transcriptions.
Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition
The external language models (LM) integration remains a challenging task for end-to-end (E2E) automatic speech recognition (ASR) which has no clear division between acoustic and language models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis
Multi-speaker speech recognition of unsegmented recordings has diverse applications such as meeting transcription and automatic subtitle generation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Don't shoot butterfly with rifles: Multi-channel Continuous Speech Separation with Early Exit Transformer
With its strong modeling capacity that comes from a multi-head and multi-layer structure, Transformer is a very powerful model for learning a sequential representation and has been successfully applied to speech separation recently.
On Minimum Word Error Rate Training of the Hybrid Autoregressive Transducer
Hybrid Autoregressive Transducer (HAT) is a recently proposed end-to-end acoustic model that extends the standard Recurrent Neural Network Transducer (RNN-T) for the purpose of the external language model (LM) fusion.
Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset
Recently, Transformer based end-to-end models have achieved great success in many areas including speech recognition.
Speaker Separation Using Speaker Inventories and Estimated Speech
We propose speaker separation using speaker inventories and estimated speech (SSUSIES), a framework leveraging speaker profiles and estimated speech for speaker separation.
An End-to-end Architecture of Online Multi-channel Speech Separation
Previously, we introduced a sys-tem, calledunmixing, fixed-beamformerandextraction(UFE), that was shown to be effective in addressing the speech over-lap problem in conversation transcription.
Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
We present a structured overview of adaptation algorithms for neural network-based speech recognition, considering both hybrid hidden Markov model / neural network systems and end-to-end neural network systems, with a focus on speaker adaptation, domain adaptation, and accent adaptation.
Continuous Speech Separation with Conformer
Continuous speech separation plays a vital role in complicated speech related tasks such as conversation transcription.
Ranked #1 on
Speech Separation
on LibriCSS
(using extra training data)
Transfer Learning Approaches for Streaming End-to-End Speech Recognition System
Transfer learning (TL) is widely used in conventional hybrid automatic speech recognition (ASR) system, to transfer the knowledge from source to target language.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability
Because of its streaming nature, recurrent neural network transducer (RNN-T) is a very promising end-to-end (E2E) model that may replace the popular hybrid model for automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Production de la parole en r\'eponse \`a de multiples perturbations du feedback auditif (Speech production in response to multiple perturbations of auditory feedback)
Des {\'e}tudes ant{\'e}rieures ont montr{\'e} que la production de la parole d{\'e}pend des conditions du feedback auditif.
On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition
Among all three E2E models, transformer-AED achieved the best accuracy in both streaming and non-streaming mode.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Exploring Transformers for Large-Scale Speech Recognition
While recurrent neural networks still largely define state-of-the-art speech recognition systems, the Transformer network has been proven to be a competitive alternative, especially in the offline condition.
Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognition
Recently, the recurrent neural network transducer (RNN-T) architecture has become an emerging trend in end-to-end automatic speech recognition research due to its advantages of being capable for online streaming speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
L-Vector: Neural Label Embedding for Domain Adaptation
We propose a novel neural label embedding (NLE) scheme for the domain adaptation of a deep neural network (DNN) acoustic model with unpaired data samples from source and target domains.
Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR
This leads to an inevitable latency during inference.
Low Latency End-to-End Streaming Speech Recognition with a Scout Network
The attention-based Transformer model has achieved promising results for speech recognition (SR) in the offline mode.
Audio and Speech Processing
High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model
While the community keeps promoting end-to-end models over conventional hybrid models, which usually are long short-term memory (LSTM) models trained with a cross entropy criterion followed by a sequence discriminative training criterion, we argue that such conventional hybrid models can still be significantly improved.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Continuous speech separation: dataset and analysis
In this paper, we define continuous speech separation (CSS) as a task of generating a set of non-overlapped speech signals from a \textit{continuous} audio stream that contains multiple utterances that are \emph{partially} overlapped by a varying degree.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Character-Aware Attention-Based End-to-End Speech Recognition
However, as one input to the decoder recurrent neural network (RNN), each WSU embedding is learned independently through context and acoustic information in a purely data-driven fashion.
Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition
In this work, we extend the T/S learning to large-scale unsupervised domain adaptation of an attention-based end-to-end (E2E) model through two levels of knowledge transfer: teacher's token posteriors as soft labels and one-best predictions as decoder guidance.
Semantic Mask for Transformer based End-to-End Speech Recognition
Attention-based encoder-decoder model has achieved impressive results for both automatic speech recognition (ASR) and text-to-speech (TTS) tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Speaker Adaptation for Attention-Based End-to-End Speech Recognition
We propose three regularization-based speaker adaptation approaches to adapt the attention-based encoder-decoder (AED) model with very limited adaptation data from target speakers for end-to-end automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Improving RNN Transducer Modeling for End-to-End Speech Recognition
In this paper, we improve the RNN-T training in two aspects.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Adversarial Speaker Adaptation
We propose a novel adversarial speaker adaptation (ASA) scheme, in which adversarial learning is applied to regularize the distribution of deep hidden features in a speaker-dependent (SD) deep neural network (DNN) acoustic model to be close to that of a fixed speaker-independent (SI) DNN acoustic model during adaptation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Adversarial Speaker Verification
The use of deep networks to extract embeddings for speaker recognition has proven successfully.
Conditional Teacher-Student Learning
To overcome this problem, we propose a conditional T/S learning scheme, in which a "smart" student model selectively chooses to learn from either the teacher model or the ground truth labels conditioned on whether the teacher can correctly predict the ground truth.
Attentive Adversarial Learning for Domain-Invariant Training
Adversarial domain-invariant training (ADIT) proves to be effective in suppressing the effects of domain variability in acoustic modeling and has led to improved performance in automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Speaker Adaptation for End-to-End CTC Models
We propose two approaches for speaker adaptation in end-to-end (E2E) automatic speech recognition systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Advancing Acoustic-to-Word CTC Model with Attention and Mixed-Units
In particular, we introduce Attention CTC, Self-Attention CTC, Hybrid CTC, and Mixed-unit CTC.
Cycle-Consistent Speech Enhancement
In this paper, we propose a cycle-consistent speech enhancement (CSE) in which an additional inverse mapping network is introduced to reconstruct the noisy features from the enhanced ones.
Adversarial Feature-Mapping for Speech Enhancement
To achieve better performance on ASR task, senone-aware (SA) AFM is further proposed in which an acoustic model network is jointly trained with the feature-mapping and discriminator networks to optimize the senone classification loss in addition to the AFM losses.
Layer Trajectory LSTM
In this paper, we propose a layer trajectory LSTM (ltLSTM) which builds a layer-LSTM using all the layer outputs from a standard multi-layer time-LSTM.
Recent Progresses in Deep Learning based Acoustic Models (Updated)
In this paper, we summarize recent progresses made in deep learning based acoustic models and the motivation and insights behind the surveyed techniques.
Developing Far-Field Speaker System Via Teacher-Student Learning
In this study, we develop the keyword spotting (KWS) and acoustic model (AM) components in a far-field speaker system.
Speaker-Invariant Training via Adversarial Learning
We propose a novel adversarial multi-task learning scheme, aiming at actively curtailing the inter-talker feature variability while maximizing its senone discriminability so as to enhance the performance of a deep neural network (DNN) based ASR system.
Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation
In this method, a student acoustic model and a condition classifier are jointly optimized to minimize the Kullback-Leibler divergence between the output distributions of the teacher and student models, and simultaneously, to min-maximize the condition classification loss.
Advancing Connectionist Temporal Classification With Attention Modeling
In this study, we propose advancing all-neural speech recognition by directly incorporating attention modeling within the Connectionist Temporal Classification (CTC) framework.
Advancing Acoustic-to-Word CTC Model
However, the word-based CTC model suffers from the out-of-vocabulary (OOV) issue as it can only model limited number of words in the output layer and maps all the remaining words into an OOV output node.
Acoustic-To-Word Model Without OOV
However, this type of word-based CTC model suffers from the out-of-vocabulary (OOV) issue as it can only model limited number of words in the output layer and maps all the remaining words into an OOV output node.
Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
Unsupervised domain adaptation of speech signal aims at adapting a well-trained source-domain acoustic model to the unlabeled data from target domain.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Improved training for online end-to-end speech recognition systems
Achieving high accuracy with end-to-end speech recognizers requires careful parameter initialization prior to training.
Large-Scale Domain Adaptation via Teacher-Student Learning
High accuracy speech recognition requires a large amount of transcribed data for supervised training.
Progressive Joint Modeling in Unsupervised Single-channel Overlapped Speech Recognition
We propose to advance the current state of the art by imposing a modular structure on the neural network, applying a progressive pretraining regimen, and improving the objective function with transfer learning and a discriminative training criterion.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
End-to-End Attention based Text-Dependent Speaker Verification
A new type of End-to-End system for text-dependent speaker verification is presented in this paper.
Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation
This work presents a broad study on the adaptation of neural network acoustic models by means of learning hidden unit contributions (LHUC) -- a method that linearly re-combines hidden units in a speaker- or environment-dependent manner using small amounts of unsupervised adaptation data.
