Search Results for author:
Found 279 papers, 90 papers with code
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units
We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization.
Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning
Recently, there has been an increasing interest in end-to-end speech recognition that directly transcribes speech to text without any predefined alignments.
Joint CTC/attention decoding for end-to-end speech recognition
End-to-end automatic speech recognition (ASR) has become a popular alternative to conventional DNN/HMM systems because it avoids the need for linguistic resources such as pronunciation dictionary, tokenization, and context-dependency trees, leading to a greatly simplified model-building process.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Multi-blank Transducers for Speech Recognition
This paper proposes a modification to RNN-Transducer (RNN-T) models for automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Efficient Sequence Transduction by Jointly Predicting Tokens and Durations
TDT models for Speech Recognition achieve better accuracy and up to 2. 82X faster inference than conventional Transducers.
Intent Classification
Intent Classification and Slot Filling
+3
Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates
We propose Fast-MD, a fast MD model that generates HI by non-autoregressive (NAR) decoding based on connectionist temporal classification (CTC) outputs followed by an ASR decoder.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
In this work, we propose a multi-modal AI system named AudioGPT, which complements LLMs (i. e., ChatGPT) with 1) foundation models to process complex audio information and solve numerous understanding and generation tasks; and 2) the input/output interface (ASR, TTS) to support spoken dialogue.
A Comparative Study on Transformer vs RNN in Speech Applications
Sequence-to-sequence models have been widely used in end-to-end speech processing, for example, automatic speech recognition (ASR), speech translation (ST), and text-to-speech (TTS).
Ranked #12 on
Speech Recognition
on AISHELL-1
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Multilingual End-to-End Speech Translation
In this paper, we propose a simple yet effective framework for multilingual end-to-end speech translation (ST), in which speech utterances in source languages are directly translated to the desired target languages with a universal sequence-to-sequence architecture.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
Furthermore, the unified design enables the integration of ASR functions with TTS, e. g., ASR-based objective evaluation and semi-supervised learning with both ASR and TTS models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
ESPnet-ST: All-in-One Speech Translation Toolkit
We present ESPnet-ST, which is designed for the quick development of speech-to-speech translation systems in a single framework.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
The Sequence-to-Sequence Baseline for the Voice Conversion Challenge 2020: Cascading ASR and TTS
This paper presents the sequence-to-sequence (seq2seq) baseline system for the voice conversion challenge (VCC) 2020.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models
Non-autoregressive (NAR) modeling has gained more and more attention in speech processing.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
ESPnet2-TTS: Extending the Edge of TTS Research
This paper describes ESPnet2-TTS, an end-to-end text-to-speech (E2E-TTS) toolkit.
ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet
However, there are few open source toolkits that can be used to generate reproducible results on different Spoken Language Understanding (SLU) benchmarks.
EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers
In this paper, we present a novel framework that jointly performs three tasks: speaker diarization, speech separation, and speaker counting.
Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation
The present work examines the assumption that combining non-learnable SF extractors to SSL models is an effective approach to low resource speech tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding
Conformer has proven to be effective in many speech processing tasks.
ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding
To showcase such integration, we performed experiments on carefully designed synthetic datasets for noisy-reverberant multi-channel ST and SLU tasks, which can be used as benchmark corpora for future research.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
E-Branchformer: Branchformer with Enhanced merging for speech recognition
Conformer, combining convolution and self-attention sequentially to capture both local and global information, has shown remarkable performance and is currently regarded as the state-of-the-art for automatic speech recognition (ASR).
Ranked #9 on
Speech Recognition
on LibriSpeech test-other
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models
End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation.
EURO: ESPnet Unsupervised ASR Open-source Toolkit
This paper describes the ESPnet Unsupervised ASR Open-source Toolkit (EURO), an end-to-end open-source toolkit for unsupervised automatic speech recognition (UASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SpeechLMScore: Evaluating speech generation using speech language model
While human evaluation is the most reliable metric for evaluating speech generation systems, it is generally costly and time-consuming.
Improving Massively Multilingual ASR With Auxiliary CTC Objectives
In this paper, we introduce our work on improving performance on FLEURS, a 102-language open ASR benchmark, by conditioning the entire model on language identity (LID).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit
ESPnet-ST-v2 is a revamp of the open-source ESPnet-ST toolkit necessitated by the broadening interests of the spoken language translation community.
A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Tasks
Conformer, a convolution-augmented Transformer variant, has become the de facto encoder architecture for speech processing due to its superior performance in various tasks, including automatic speech recognition (ASR), speech translation (ST) and spoken language understanding (SLU).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A New Benchmark of Aphasia Speech Recognition and Detection Based on E-Branchformer and Multi-task Learning
Our system achieves state-of-the-art speaker-level detection accuracy (97. 3%), and a relative WER reduction of 11% for moderate Aphasia patients.
Bayes Risk Transducer: Transducer with Controllable Alignment Prediction
While the vanilla transducer does not have a prior preference for any of the valid paths, this work intends to enforce the preferred paths and achieve controllable alignment prediction.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
Pre-training speech models on large volumes of data has achieved remarkable success.
Improving Design of Input Condition Invariant Speech Enhancement
In this paper we propose novel architectures to improve the input condition invariant SE model so that performance in simulated conditions remains competitive while real condition degradation is much mitigated.
ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models
First, we provide an open-source platform for researchers in the speaker recognition community to effortlessly build models.
Ranked #1 on
Speaker Verification
on VoxCeleb
(using extra training data)
Arabic Speech Recognition by End-to-End, Modular Systems and Human
Recent advances in automatic speech recognition (ASR) have achieved accuracy levels comparable to human transcribers, which led researchers to debate if the machine has reached human performance.
SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition
In the English speech-to-text (STT) machine learning task, acoustic models are conventionally trained on uncased Latin characters, and any necessary orthography (such as capitalization, punctuation, and denormalization of non-standard words) is imputed by separate post-processing models.
Ranked #3 on
Speech Recognition
on SPGISpeech
Unsupervised Data Selection for TTS: Using Arabic Broadcast News as a Case Study
Several high-resource Text to Speech (TTS) systems currently produce natural, well-established human-like speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
TorchAudio: Building Blocks for Audio and Speech Processing
This document describes version 0. 10 of TorchAudio: building blocks for machine learning applications in the audio and speech processing domain.
TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch
TorchAudio is an open-source audio and speech processing library built for PyTorch.
Deep clustering: Discriminative embeddings for segmentation and separation
The framework can be used without class labels, and therefore has the potential to be trained on a diverse set of sound types, and to generalize to novel sources.
Ranked #30 on
Speech Separation
on WSJ0-2mix
SUPERB: Speech processing Universal PERformance Benchmark
SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data.
S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations
In this work, we provide a series of in-depth analyses by benchmarking on the two tasks in VCC2020, namely intra-/cross-lingual any-to-one (A2O) VC, as well as an any-to-any (A2A) setting.
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities
In this paper, we introduce SUPERB-SG, a new benchmark focused on evaluating the semantic and generative capabilities of pre-trained models by increasing task diversity and difficulty over SUPERB.
Self-supervised Representation Learning for Speech Processing
Due to the growing popularity of SSL, and the shared mission of the areas in bringing speech and language technologies to more use cases with better quality and scaling the technologies for under-represented languages, we propose this tutorial to systematically survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing.
Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM
The CTC network sits on top of the encoder and is jointly trained with the attention-based decoder.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
We present Espresso, an open-source, modular, extensible end-to-end neural automatic speech recognition (ASR) toolkit based on the deep learning library PyTorch and the popular neural machine translation toolkit fairseq.
Ranked #1 on
Speech Recognition
on Hub5'00 CallHome
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio
This paper introduces GigaSpeech, an evolving, multi-domain English speech recognition corpus with 10, 000 hours of high quality labeled audio suitable for supervised training, and 40, 000 hours of total audio suitable for semi-supervised and unsupervised training.
Ranked #1 on
Speech Recognition
on GigaSpeech
End-to-End Neural Speaker Diarization with Self-attention
Our method was even better than that of the state-of-the-art x-vector clustering-based method.
Ranked #2 on
Speaker Diarization
on CALLHOME
End-to-End Neural Speaker Diarization with Permutation-Free Objectives
To realize such a model, we formulate the speaker diarization problem as a multi-label classification problem, and introduces a permutation-free objective function to directly minimize diarization errors without being suffered from the speaker-label permutation problem.
Ranked #6 on
Speaker Diarization
on CALLHOME
End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors
End-to-end speaker diarization for an unknown number of speakers is addressed in this paper.
Neural Speaker Diarization with Speaker-Wise Chain Rule
Speaker diarization is an essential step for processing multi-speaker audio.
A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech
Recent Text-to-Speech (TTS) systems trained on reading or acted corpora have achieved near human-level naturalness.
A Study on Speech Enhancement Based on Diffusion Probabilistic Model
Based on this property, we propose a diffusion probabilistic model-based speech enhancement (DiffuSE) model that aims to recover clean speech signals from noisy signals.
Conditional Diffusion Probabilistic Model for Speech Enhancement
Speech enhancement is a critical component of many user-oriented audio applications, yet current systems still suffer from distorted and unnatural outputs.
ESPnet-ONNX: Bridging a Gap Between Research and Production
In the field of deep learning, researchers often focus on inventing novel neural network models and improving benchmarks.
Zero-shot Learning for Grapheme to Phoneme Conversion with Language Ensemble
Grapheme-to-Phoneme (G2P) has many applications in NLP and speech fields.
Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization
We investigate the emergent abilities of the recently proposed web-scale speech model Whisper, by adapting it to unseen tasks with prompt engineering.
Single-Channel Multi-Speaker Separation using Deep Clustering
In this paper we extend the baseline system with an end-to-end signal approximation objective that greatly improves performance on a challenging speech separation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
To achieve comprehensive coverage of diverse speech tasks and harness instruction tuning, we invite the community to collaborate and contribute, facilitating the dynamic growth of the benchmark.
End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification
However, the clustering-based approach has a number of problems; i. e., (i) it is not optimized to minimize diarization errors directly, (ii) it cannot handle speaker overlaps correctly, and (iii) it has trouble adapting their speaker embedding models to real audio recordings with speaker overlaps.
Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding
Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow.
DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
Knowledge distillation trains a small student model to mimic the behavior of a large teacher model.
Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages
We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data.
Ranked #3 on
Named Entity Recognition (NER)
on SLUE
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
INTERSPEECH 2021 ConferencingSpeech Challenge: Towards Far-field Multi-Channel Speech Enhancement for Video Conferencing
The ConferencingSpeech 2021 challenge is proposed to stimulate research on far-field multi-channel speech enhancement for video conferencing.
DOVER-Lap: A Method for Combining Overlap-aware Diarization Outputs
Several advances have been made recently towards handling overlapping speech for speaker diarization.
Audio and Speech Processing Sound
A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units
To address these issues, we devise a cascaded modular system leveraging self-supervised discrete speech units as language representation.
PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech Enhancement
We can add this criterion as an auxiliary loss to any model that produces speech, to optimize speech outputs to match the values of clean speech in these features.
TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement
We propose an objective for perceptual quality based on temporal acoustic parameters.
Speaker Diarization with Region Proposal Network
Speaker diarization is an important pre-processing step for many speech applications, and it aims to solve the "who spoke when" problem.
Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining
While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data.
HEAR: Holistic Evaluation of Audio Representations
The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios.
ASR2K: Speech Recognition for Around 2000 Languages without Audio
We achieve 50% CER and 74% WER on the Wilderness dataset with Crubadan statistics only and improve them to 45% CER and 69% WER when using 10000 raw text utterances.
Deep Speech Synthesis from Articulatory Representations
In the articulatory synthesis task, speech is synthesized from input features containing information about the physical behavior of the human vocal tract.
Speaker-Independent Acoustic-to-Articulatory Speech Inversion
To build speech processing methods that can handle speech as naturally as humans, researchers have explored multiple ways of building an invertible mapping from speech to an interpretable space.
Deep Speech Synthesis from MRI-Based Articulatory Representations
Finally, through a series of ablations, we show that the proposed MRI representation is more comprehensive than EMA and identify the most suitable MRI feature subset for articulatory synthesis.
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information.
Weakly Supervised Deep Recurrent Neural Networks for Basic Dance Step Generation
However, applying DNNs for generating dance to a piece of music is nevertheless challenging, because of 1) DNNs need to generate large sequences while mapping the music input, 2) the DNN needs to constraint the motion beat to the music, and 3) DNNs require a considerable amount of hand-crafted data.
Improving Speech Enhancement through Fine-Grained Speech Characteristics
We first identify key acoustic parameters that have been found to correlate well with voice quality (e. g. jitter, shimmer, and spectral flux) and then propose objective functions which are aimed at reducing the difference between clean speech and enhanced speech with respect to these features.
InterMPL: Momentum Pseudo-Labeling with Intermediate CTC Loss
This paper presents InterMPL, a semi-supervised learning method of end-to-end automatic speech recognition (ASR) that performs pseudo-labeling (PL) with intermediate supervision.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Speech collage: code-switched audio generation by collaging monolingual corpora
Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources.
Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain
Moreover, by including the data of variable numbers of speakers, our model can even better than the PIT-Conformer AR model with only 1/7 latency, obtaining WERs of 19. 9% and 34. 3% on WSJ0-2mix and WSJ0-3mix sets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages
We propose a novel Tri-Modal Translation (TMT) model that translates between arbitrary modalities spanning speech, image, and text.
Learning Speaker Embedding from Text-to-Speech
TTS with speaker classification loss improved EER by 0. 28\% and 0. 73\% absolutely from a model using only speaker classification loss in LibriTTS and Voxceleb1 respectively.
Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
As users increasingly rely on cloud-based computing services, it is important to ensure that uploaded speech data remains private.
Attention-based Multi-hypothesis Fusion for Speech Summarization
We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition
Self-supervised ASR-TTS models suffer in out-of-domain data conditions.
Differentiable Allophone Graphs for Language-Universal Speech Recognition
These phone-based systems with learned allophone graphs can be used by linguists to document new languages, build phone-based lexicons that capture rich pronunciation variations, and re-evaluate the allophone mappings of seen language.
When Is TTS Augmentation Through a Pivot Language Useful?
Developing Automatic Speech Recognition (ASR) for low-resource languages is a challenge due to the small amount of transcribed audio data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Understanding Probe Behaviors through Variational Bounds of Mutual Information
With the success of self-supervised representations, researchers seek a better understanding of the information encapsulated within a representation.
AugSumm: towards generalizable speech summarization using synthetic labels from large language model
We tackle this challenge by proposing AugSumm, a method to leverage large language models (LLMs) as a proxy for human annotators to generate augmented summaries for training and evaluation.
Cross-lingual Transfer for Speech Processing using Acoustic Language Similarity
We demonstrate the effectiveness of our approach in language family classification, speech recognition, and speech synthesis tasks.
Multi-Modal Data Augmentation for End-to-End ASR
We present a new end-to-end architecture for automatic speech recognition (ASR) that can be trained using \emph{symbolic} input in addition to the traditional acoustic input.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
A Purely End-to-end System for Multi-speaker Speech Recognition
In this paper, we propose a new sequence-to-sequence framework to directly decode multiple label sequences from a single speech sequence by unifying source separation and speech recognition functions in an end-to-end manner.
Multi-Head Decoder for End-to-End Speech Recognition
This paper presents a new network architecture called multi-head decoder for end-to-end speech recognition as an extension of a multi-head attention model.
ESPnet: End-to-End Speech Processing Toolkit
This paper introduces a new open source platform for end-to-end speech processing named ESPnet.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines
The CHiME challenge series aims to advance robust automatic speech recognition (ASR) technology by promoting research at the interface of speech and language processing, signal processing , and machine learning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Building state-of-the-art distant speech recognition using the CHiME-4 challenge with a setup of speech enhancement baseline
This paper describes a new baseline system for automatic speech recognition (ASR) in the CHiME-4 challenge to promote the development of noisy ASR in speech processing communities by providing 1) state-of-the-art system with a simplified single system comparable to the complicated top systems in the challenge, 2) publicly available and reproducible recipe through the main repository in the Kaldi speech recognition toolkit.
Ranked #2 on
Noisy Speech Recognition
on CHiME real
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition
Further, we use hidden units in the deep LSTM acoustic model to assist in predicting the beamforming filter coefficients.
Multichannel End-to-end Speech Recognition
The field of speech recognition is in the midst of a paradigm shift: end-to-end neural networks are challenging the dominance of hidden Markov models as a core technology.
Low-Resource Contextual Topic Identification on Speech
In topic identification (topic ID) on real-world unstructured audio, an audio instance of variable topic shifts is first broken into sequential segments, and each segment is independently classified.
Back-Translation-Style Data Augmentation for End-to-End ASR
In this paper we propose a novel data augmentation method for attention-based end-to-end automatic speech recognition (E2E-ASR), utilizing a large amount of text which is not paired with speech signals.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
End-to-end Speech Recognition with Word-based RNN Language Models
This paper investigates the impact of word-based RNN language models (RNN-LMs) on the performance of end-to-end automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Phasebook and Friends: Leveraging Discrete Representations for Source Separation
Here, we propose "magbook", "phasebook", and "combook", three new types of layers based on discrete representations that can be used to estimate complex time-frequency masks.
Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling
In this work, we attempt to use data from 10 BABEL languages to build a multi-lingual seq2seq model as a prior model, and then port them towards 4 other BABEL languages using transfer learning approach.
Language Modelling
Sequence-To-Sequence Speech Recognition
+2
Cycle-consistency training for end-to-end speech recognition
To solve this problem, this work presents a loss that is based on the speech encoder state sequence instead of the raw speech signal.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Transfer learning of language-independent end-to-end ASR with language model fusion
This work explores better adaptation methods to low-resource languages using an external language model (LM) under the framework of transfer learning.
End-to-End Monaural Multi-speaker ASR System without Pretraining
The experiments demonstrate that the proposed methods can improve the performance of the end-to-end model in separating the overlapping speech and recognizing the separated streams.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Promising Accurate Prefix Boosting for sequence-to-sequence ASR
In this paper, we present promising accurate prefix boosting (PAPB), a discriminative training technique for attention based sequence-to-sequence (seq2seq) ASR.
CNN-based MultiChannel End-to-End Speech Recognition for everyday home environments
By employing a convolutional neural network (CNN)-based multichannel end-to-end speech recognition system, this study attempts to overcome the presents difficulties in everyday environments.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Building Corpora for Single-Channel Speech Separation Across Multiple Domains
To date, the bulk of research on single-channel speech separation has been conducted using clean, near-field, read speech, which is not representative of many modern applications.
Analysis of Multilingual Sequence-to-Sequence speech recognition systems
This paper investigates the applications of various multilingual approaches developed in conventional hidden Markov model (HMM) systems to sequence-to-sequence (seq2seq) automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Stream attention-based multi-array end-to-end speech recognition
Automatic Speech Recognition (ASR) using multiple microphone arrays has achieved great success in the far-field robustness.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Multi-encoder multi-resolution framework for end-to-end speech recognition
In this work, we present a novel Multi-Encoder Multi-Resolution (MEMR) framework based on the joint CTC/Attention model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Vectorization of hypotheses and speech for faster beam search in encoder decoder-based speech recognition
In this paper, we propose a parallelism technique for beam search, which accelerates the search process by vectorizing multiple hypotheses to eliminate the for-loop program.
Improving End-to-end Speech Recognition with Pronunciation-assisted Sub-word Modeling
Most end-to-end speech recognition systems model text directly as a sequence of characters or sub-words.
Pretraining by Backtranslation for End-to-end ASR in Low-Resource Settings
Using transcribed speech from nearby languages gives a further 20-30% relative reduction in character error rate.
Massively Multilingual Adversarial Speech Recognition
We report on adaptation of multilingual end-to-end speech recognition models trained on as many as 100 languages.
An Investigation of End-to-End Multichannel Speech Recognition for Reverberant and Mismatch Conditions
This report investigates the ability of E2E ASR from standard close-talk to far-field applications by encompassing entire multichannel speech enhancement and ASR components within the S2S model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text
Such techniques derive training procedures and losses able to leverage unpaired speech and/or text data by combining ASR with Text-to-Speech (TTS) models.
Ranked #33 on
Semi-Supervised Image Classification
on ImageNet - 10% labeled data
(Top 5 Accuracy metric)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Multi-Stream End-to-End Speech Recognition
Two representative framework have been proposed and discussed, which are Multi-Encoder Multi-Resolution (MEM-Res) framework and Multi-Encoder Multi-Array (MEM-Array) framework, respectively.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognition
In this paper, we propose a novel auxiliary loss function for target-speaker automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Simultaneous Speech Recognition and Speaker Diarization for Monaural Dialogue Recordings with Target-Speaker Acoustic Models
Our proposed method combined with i-vector speaker embeddings ultimately achieved a WER that differed by only 2. 1 % from that of TS-ASR given oracle speaker embeddings.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition
In this work, we propose a novel neural sequence-to-sequence (seq2seq) architecture, MIMO-Speech, which extends the original seq2seq to deal with multi-channel input and multi-channel output so that it can fully model multi-channel multi-speaker speech separation and recognition.
Transformer ASR with Contextual Block Processing
In this paper, we propose a new block processing method for the Transformer encoder by introducing a context-aware inheritance mechanism.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A practical two-stage training strategy for multi-stream end-to-end speech recognition
The multi-stream paradigm of audio processing, in which several sources are simultaneously considered, has been an active research area for information fusion.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Towards Online End-to-end Transformer Automatic Speech Recognition
In this paper, we extend it towards an entire online E2E ASR system by introducing an online decoding process inspired by monotonic chunkwise attention (MoChA) into the Transformer decoder.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognition
In this paper, we study two different non-autoregressive transformer structure for automatic speech recognition (ASR): A-CMLM and A-FMLM.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Sequential Multi-Frame Neural Beamforming for Speech Separation and Enhancement
This work introduces sequential neural beamforming, which alternates between neural network based spectral separation and beamforming based spatial separation.
End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection
The proposed method is publicly available.
End-to-End Multi-speaker Speech Recognition with Transformer
Recently, fully recurrent neural network (RNN) based end-to-end models have been proven to be effective for multi-speaker speech recognition in both the single-channel and multi-channel scenarios.
CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings
Following the success of the 1st, 2nd, 3rd, 4th and 5th CHiME challenges we organize the 6th CHiME Speech Separation and Recognition Challenge (CHiME-6).
DiscreTalk: Text-to-Speech as a Machine Translation Problem
This paper proposes a new end-to-end text-to-speech (E2E-TTS) model based on neural machine translation (NMT).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict
In this work, Mask CTC model is trained using a Transformer encoder-decoder with joint training of mask prediction and CTC.
Audio and Speech Processing Sound
Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals
This model additionally has a simple and efficient stop criterion for the end of the transduction, making it able to infer the variable number of output sequences.
Ranked #3 on
Speech Separation
on WSJ0-4mix
Speaker-Conditional Chain Model for Speech Separation and Extraction
With the predicted speaker information from whole observation, our model is helpful to solve the problem of conventional speech separation and speaker extraction for multi-round long recordings.
Audio and Speech Processing Sound
Insertion-Based Modeling for End-to-End Automatic Speech Recognition
One NAT model, mask-predict, has been applied to ASR but the model needs some heuristics or additional component to estimate the length of the output token sequence.
Audio and Speech Processing Sound
Augmentation adversarial training for self-supervised speaker recognition
Since the augmentation simulates the acoustic characteristics, training the network to be invariant to augmentation also encourages the network to be invariant to the channel information in general.
Streaming Transformer ASR with Blockwise Synchronous Inference
In this paper, we extend block processing towards an entire streaming E2E ASR system without additional training, by introducing a blockwise synchronous decoding process inspired by a neural transducer into the Transformer decoder.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Improved Mask-CTC for Non-Autoregressive End-to-End ASR
While Mask-CTC achieves remarkably fast inference speed, its recognition performance falls behind that of conventional autoregressive (AR) systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder
Fast inference speed is an important goal towards real-world deployment of speech translation (ST) systems.
Directional ASR: A New Paradigm for E2E Multi-Speaker Speech Recognition with Source Localization
The advantages of D-ASR over existing methods are threefold: (1) it provides explicit speaker locations, (2) it improves the explainability factor, and (3) it achieves better ASR performance as the process is more streamlined.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis
Multi-speaker speech recognition of unsegmented recordings has diverse applications such as meeting transcription and automatic subtitle generation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Improving Speech Enhancement Performance by Leveraging Contextual Broad Phonetic Class Information
Experimental results from speech denoising, speech dereverberation, and impaired speech enhancement tasks confirmed that contextual BPC information improves SE performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Continuous Speech Separation Using Speaker Inventory for Long Multi-talker Recording
Leveraging additional speaker information to facilitate speech separation has received increasing attention in recent years.
End-to-End Speaker Diarization as Post-Processing
Clustering-based diarization methods partition frames into clusters of the number of speakers; thus, they typically cannot handle overlapping speech because each frame is assigned to one speaker.
Online Streaming End-to-End Neural Diarization Handling Overlapping Speech and Flexible Numbers of Speakers
We propose a streaming diarization method based on an end-to-end neural diarization (EEND) model, which handles flexible numbers of speakers and overlapping speech.
Speaker Diarization
Sound
Audio and Speech Processing
A Review of Speaker Diarization: Recent Advances with Deep Learning
Speaker diarization is a task to label audio or video recordings with classes that correspond to speaker identity, or in short, a task to identify "who spoke when".
Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yoloxóchitl Mixtec
"Transcription bottlenecks", created by a shortage of effective human transcribers are one of the main challenges to endangered language (EL) documentation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
The Hitachi-JHU DIHARD III System: Competitive End-to-End Neural Diarization and X-Vector Clustering Systems Combined by DOVER-Lap
This paper provides a detailed description of the Hitachi-JHU system that was submitted to the Third DIHARD Speech Diarization Challenge.
Intermediate Loss Regularization for CTC-based Speech Recognition
In addition, we propose to combine this intermediate CTC loss with stochastic depth training, and apply this combination to a recently proposed Conformer network.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Improving RNN Transducer With Target Speaker Extraction and Neural Uncertainty Estimation
Target-speaker speech recognition aims to recognize target-speaker speech from noisy environments with background noise and interfering speakers.
Speech Enhancement
Speech Extraction
+1
Sound
Audio and Speech Processing
Online End-to-End Neural Diarization with Speaker-Tracing Buffer
This paper proposes a novel online speaker diarization algorithm based on a fully supervised self-attention mechanism (SA-EEND).
Dual-Path Modeling for Long Recording Speech Separation in Meetings
A transformer-based dual-path system is proposed, which integrates transform layers for global modeling.
End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend
Recently, the end-to-end approach has been successfully applied to multi-speaker speech separation and recognition in both single-channel and multichannel conditions.
Gaussian Kernelized Self-Attention for Long Sequence Data and Its Application to CTC-based Speech Recognition
Self-attention (SA) based models have recently achieved significant performance improvements in hybrid and end-to-end automatic speech recognition (ASR) systems owing to their flexible context modeling capability.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Deep Learning based Multi-Source Localization with Source Splitting and its Effectiveness in Multi-Talker Speech Recognition
In addition to using the prediction error as a metric for evaluating our localization model, we also establish its potency as a frontend with automatic speech recognition (ASR) as the downstream task.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation
To leverage the full potential of the source language information, we propose backward SeqKD, SeqKD from a target-to-source backward NMT model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
In this work, we present an end-to-end framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks.
End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings
We present an end-to-end deep network model that performs meeting diarization from single-channel audio recordings.
Self-Guided Curriculum Learning for Neural Machine Translation
In the field of machine learning, the well-trained model is assumed to be able to recover the training labels, i. e. the synthetic labels predicted by the model should be as close to the ground-truth labels as possible.
Data Augmentation Methods for End-to-end Speech Recognition on Distant-Talk Scenarios
Although end-to-end automatic speech recognition (E2E ASR) has achieved great performance in tasks that have numerous paired data, it is still challenging to make E2E ASR robust against noisy and low-resource conditions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
End-to-End Speaker Diarization Conditioned on Speech Activity and Overlap Detection
In this paper, we present a conditional multitask learning method for end-to-end neural speaker diarization (EEND).
Semi-Supervised Training with Pseudo-Labeling for End-to-End Neural Diarization
To evaluate our proposed method, we conduct the experiments of model adaptation using labeled and unlabeled data.
Leveraging Pre-trained Language Model for Speech Sentiment Analysis
In this paper, we explore the use of pre-trained language models to learn sentiment information of written texts for speech sentiment analysis.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
End-to-end ASR to jointly predict transcriptions and linguistic annotations
We propose a Transformer-based sequence-to-sequence model for automatic speech recognition (ASR) capable of simultaneously transcribing and annotating audio with linguistic information such as phonemic transcripts or part-of-speech (POS) tags.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yol\'oxochitl Mixtec
{``}Transcription bottlenecks{''}, created by a shortage of effective human transcribers (i. e., transcriber shortage), are one of the main challenges to endangered language (EL) documentation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Layer Pruning on Demand with Intermediate CTC
Deploying an end-to-end automatic speech recognition (ASR) model on mobile/embedded devices is a challenging task, since the device computational power and energy consumption requirements are dynamically changed in practice.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Multi-mode Transformer Transducer with Stochastic Future Context
A Multi-mode ASR model can fulfill various latency requirements during inference -- when a larger latency becomes acceptable, the model can process longer future context to achieve higher accuracy and when a latency budget is not flexible, the model can be less dependent on future context but still achieve reliable accuracy.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Encoder-Decoder Based Attractors for End-to-End Neural Diarization
Diarization results are then estimated as dot products of the attractors and embeddings.
Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding
Our splits identify performance gaps up to 10% between end-to-end systems that were within 1% of each other on the original test sets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
ESPnet-ST IWSLT 2021 Offline Speech Translation System
This year we made various efforts on training data, architecture, and audio segmentation.
Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors
This makes it possible to produce diarization results of a large number of speakers for the whole recording even if the number of output speakers for each subsequence is limited.
Toward Streaming ASR with Non-Autoregressive Insertion-based Model
We propose a system to concatenate audio segmentation and non-autoregressive ASR to realize high accuracy and low RTF ASR.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
On Prosody Modeling for ASR+TTS based Voice Conversion
In voice conversion (VC), an approach showing promising results in the latest voice conversion challenge (VCC) 2020 is to first use an automatic speech recognition (ASR) model to transcribe the source speech into the underlying linguistic contents; these are then used as input by a text-to-speech (TTS) system to generate the converted speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Target-speaker Voice Activity Detection with Improved I-Vector Estimation for Unknown Number of Speaker
Target-speaker voice activity detection (TS-VAD) has recently shown promising results for speaker diarization on highly overlapped speech.
Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
We propose a unified NAR E2E-ST framework called Orthros, which has an NAR decoder and an auxiliary shallow AR decoder on top of the shared encoder.
A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation
Non-autoregressive (NAR) models simultaneously generate multiple outputs in a sequence, which significantly reduces the inference speed at the cost of accuracy drop compared to autoregressive baselines.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Multi-Channel End-to-End Neural Diarization with Distributed Microphones
With simulated and real-recorded datasets, we demonstrated that the proposed method outperformed conventional EEND when a multi-channel input was given while maintaining comparable performance with a single-channel input.
An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition
The Transformer architecture has been well adopted as a dominant architecture in most sequence transduction tasks including automatic speech recognition (ASR), since its attention mechanism excels in capturing long-range dependencies.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions
The deep learning based time-domain models, e. g. Conv-TasNet, have shown great potential in both single-channel and multi-channel speech enhancement.
Sequence Transduction with Graph-based Supervision
The recurrent neural network transducer (RNN-T) objective plays a major role in building today's best automatic speech recognition (ASR) systems for production.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Highland Puebla Nahuatl Speech Translation Corpus for Endangered Language Documentation
Documentation of endangered languages (ELs) has become increasingly urgent as thousands of languages are on the verge of disappearing by the end of the 21st century.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
End-to-End Multi-Lingual Multi-Speaker Speech Recognition
Several multi-lingual ASR systems were recently proposed based on a monolithic neural network architecture without language-dependent modules, showing that modeling of multiple languages is well within the capabilities of an end-to-end framework.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorization
Conversational bilingual speech encompasses three types of utterances: two purely monolingual types and one intra-sententially code-switched type.
Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem
Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols, and convert the paradigm of the speech separation/enhancement related tasks from regression to classification.
A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies
In this study, we present recent developments of models trained with the RNN-T loss in ESPnet.
Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language models
The proposed NAR model significantly surpasses previous NAR systems on the AISHELL-1 benchmark and shows a potential for English tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Run-and-back stitch search: novel block synchronous decoding for streaming encoder-decoder ASR
To this end, we propose a novel blockwise synchronous decoding algorithm with a hybrid approach that combines endpoint prediction and endpoint post-determination.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Joint Speech Recognition and Audio Captioning
A major hurdle in evaluating our proposed approach is the lack of labeled audio datasets with both speech transcriptions and audio captions.
Acoustic Event Detection with Classifier Chains
In each iteration, the event's activity is estimated and used to condition the next output based on the probabilistic chain rule to form classifier chains.
ESPnet How2 Speech Translation System for IWSLT 2019: Pre-training, Knowledge Distillation, and Going Deeper
In this year, we mainly build our systems based on Transformer architectures in all tasks and focus on the end-to-end speech translation (E2E-ST).
Towards Low-distortion Multi-channel Speech Enhancement: The ESPNet-SE Submission to The L3DAS22 Challenge
This paper describes our submission to the L3DAS22 Challenge Task 1, which consists of speech enhancement with 3D Ambisonic microphones.
The JHU/KyotoU Speech Translation System for IWSLT 2018
This paper describes the Johns Hopkins University (JHU) and Kyoto University submissions to the Speech Translation evaluation campaign at IWSLT2018.
Extended Graph Temporal Classification for Multi-Speaker End-to-End ASR
As an example application, we use the extended GTC (GTC-e) for the multi-speaker speech recognition task.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Investigating self-supervised learning for speech enhancement and separation
Speech enhancement and separation are two fundamental tasks for robust speech processing.
SingAug: Data Augmentation for Singing Voice Synthesis with Cycle-consistent Training Strategy
Deep learning based singing voice synthesis (SVS) systems have been demonstrated to flexibly generate singing with better qualities, compared to conventional statistical parametric based methods.
Memory-Efficient Training of RNN-Transducer with Sampled Softmax
RNN-Transducer has been one of promising architectures for end-to-end automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representation
This work presents our end-to-end (E2E) automatic speech recognition (ASR) model targetting at robust speech recognition, called Integraded speech Recognition with enhanced speech Input for Self-supervised learning representation (IRIS).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
End-to-End Multi-speaker ASR with Independent Vector Analysis
We develop an end-to-end system for multi-channel, multi-speaker automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Better Intermediates Improve CTC Inference
This paper proposes a method for improved CTC inference with searched intermediates and multi-pass conditioning.
Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation
Although Transformers have gained success in several speech processing tasks like spoken language understanding (SLU) and speech translation (ST), achieving online processing while keeping competitive performance is still essential for real-world interaction.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
