Search Results for author:
Found 279 papers, 90 papers with code
The JHU/KyotoU Speech Translation System for IWSLT 2018
This paper describes the Johns Hopkins University (JHU) and Kyoto University submissions to the Speech Translation evaluation campaign at IWSLT2018.
ESPnet How2 Speech Translation System for IWSLT 2019: Pre-training, Knowledge Distillation, and Going Deeper
In this year, we mainly build our systems based on Transformer architectures in all tasks and focus on the end-to-end speech translation (E2E-ST).
CMU’s IWSLT 2022 Dialect Speech Translation System
We use additional paired Modern Standard Arabic data (MSA) to directly improve the speech recognition (ASR) and machine translation (MT) components of our cascaded systems.
Self-supervised Representation Learning for Speech Processing
Due to the growing popularity of SSL, and the shared mission of the areas in bringing speech and language technologies to more use cases with better quality and scaling the technologies for under-represented languages, we propose this tutorial to systematically survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing.
Phone Inventories and Recognition for Every Language
Identifying phone inventories is a crucial component in language documentation and the preservation of endangered languages.
Highland Puebla Nahuatl Speech Translation Corpus for Endangered Language Documentation
Documentation of endangered languages (ELs) has become increasingly urgent as thousands of languages are on the verge of disappearing by the end of the 21st century.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Findings of the IWSLT 2022 Evaluation Campaign
The evaluation campaign of the 19th International Conference on Spoken Language Translation featured eight shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Speech to speech translation, (iv) Low-resource speech translation, (v) Multilingual speech translation, (vi) Dialect speech translation, (vii) Formality control for speech translation, (viii) Isometric speech translation.
Zero-shot Learning for Grapheme to Phoneme Conversion with Language Ensemble
Grapheme-to-Phoneme (G2P) has many applications in NLP and speech fields.
A Large-Scale Evaluation of Speech Foundation Models
In this work, we establish the Speech processing Universal PERformance Benchmark (SUPERB) to study the effectiveness of the paradigm for speech.
LV-CTC: Non-autoregressive ASR with CTC and latent variable models
In this paper, we propose a new model combining CTC and a latent variable model, which is one of the state-of-the-art models in the neural machine translation research field.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Wav2Gloss: Generating Interlinear Glossed Text from Speech
Thousands of the world's languages are in danger of extinction--a tremendous threat to cultural identities and human language diversity.
Aligning Speech to Languages to Enhance Code-switching Speech Recognition
Performance evaluation using large language models reveals the advantage of the linguistic hint by achieving 14. 1% and 5. 5% relative improvement on test sets of the ASRU and SEAME datasets, respectively.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages
We propose a novel Tri-Modal Translation (TMT) model that translates between arbitrary modalities spanning speech, image, and text.
OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification
Inspired by the Open Whisper-style Speech Model (OWSM) project, we propose OWSM-CTC, a novel encoder-only speech foundation model based on Connectionist Temporal Classification (CTC).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Evaluating and Improving Continual Learning in Spoken Language Understanding
In this work, we propose an evaluation methodology that provides a unified evaluation on stability, plasticity, and generalizability in continual learning.
SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition
Existing speech language models typically utilize task-dependent prompt tokens to unify various speech tasks in a single model.
ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models
First, we provide an open-source platform for researchers in the speaker recognition community to effortlessly build models.
Ranked #1 on
Speaker Verification
on VoxCeleb
(using extra training data)
OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
In this work, we aim to improve the performance and efficiency of OWSM without extra training data.
Improving Design of Input Condition Invariant Speech Enhancement
In this paper we propose novel architectures to improve the input condition invariant SE model so that performance in simulated conditions remains competitive while real condition degradation is much mitigated.
Boosting Unknown-number Speaker Separation with Transformer Decoder-based Attractor
We propose a novel speech separation model designed to separate mixtures with an unknown number of speakers.
Ranked #1 on
Speech Separation
on WSJ0-5mix
Contextualized Automatic Speech Recognition with Attention-Based Bias Phrase Boosted Beam Search
The proposed method can be trained effectively by combining a bias phrase index loss and special tokens to detect the bias phrases in the input speech data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Improving ASR Contextual Biasing with Guided Attention
Compared to studies with similar motivations, the proposed loss operates directly on the cross attention weights and is easier to implement.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
AugSumm: towards generalizable speech summarization using synthetic labels from large language model
We tackle this challenge by proposing AugSumm, a method to leverage large language models (LLMs) as a proxy for human annotators to generate augmented summaries for training and evaluation.
Generative Context-aware Fine-tuning of Self-supervised Speech Models
Considering the recent advances in generative large language models (LLM), we hypothesize that an LLM could generate useful context information using the preceding text.
Understanding Probe Behaviors through Variational Bounds of Mutual Information
With the success of self-supervised representations, researchers seek a better understanding of the information encapsulated within a representation.
Phoneme-aware Encoding for Prefix-tree-based Contextual ASR
While the original TCPGen relies on grapheme-based encoding, we propose extending it with phoneme-aware encoding to better recognize words of unusual pronunciations.
TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch
TorchAudio is an open-source audio and speech processing library built for PyTorch.
A Single Speech Enhancement Model Unifying Dereverberation, Denoising, Speaker Counting, Separation, and Extraction
We propose a multi-task universal speech enhancement (MUSE) model that can perform five speech enhancement (SE) tasks: dereverberation, denoising, speech separation (SS), target speaker extraction (TSE), and speaker counting.
Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond
The 2023 Multilingual Speech Universal Performance Benchmark (ML-SUPERB) Challenge expands upon the acclaimed SUPERB framework, emphasizing self-supervised models in multilingual speech recognition and language identification.
HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Model
In this paper, we propose a new approach to enrich the semantic representation of HuBERT.
UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions
Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model's behavior and surpassing performance of task-specific models.
Ranked #1 on
Spoken Language Understanding
on Fluent Speech Commands
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition
This paper presents a novel framework for joint speaker diarization (SD) and automatic speech recognition (ASR), named SLIDAR (sliding-window diarization-augmented recognition).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Toward Universal Speech Enhancement for Diverse Input Conditions
Currently, there is no universal SE approach that can effectively handle diverse input conditions with a single model.
Speech collage: code-switched audio generation by collaging monolingual corpora
Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources.
Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling.
Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing
Recent works in end-to-end speech-to-text translation (ST) have proposed multi-tasking methods with soft parameter sharing which leverage machine translation (MT) data via secondary encoders that map text inputs to an eventual cross-modal representation.
Enhancing End-to-End Conversational Speech Translation Through Target Language Context Utilization
Incorporating longer context has been shown to benefit machine translation, but the inclusion of context in end-to-end speech translation (E2E-ST) remains under-studied.
Segment-Level Vectorized Beam Search Based on Partially Autoregressive Inference
Attention-based encoder-decoder models with autoregressive (AR) decoding have proven to be the dominant approach for automatic speech recognition (ASR) due to their superior accuracy.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learning
We show that further efficiency can be achieved with a vanilla HuBERT Base model, which can maintain 94% of XLS-R's performance with only 3% of the data, 4 GPUs, and limited trials.
Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
Pre-training speech models on large volumes of data has achieved remarkable success.
Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff
Further, this method lacks mechanisms for \textit{controlling} the quality vs. latency tradeoff.
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information.
Semi-Autoregressive Streaming ASR With Label Context
Non-autoregressive (NAR) modeling has gained significant interest in speech processing since these models achieve dramatically lower inference time than autoregressive (AR) models while also achieving good transcription accuracy.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
To achieve comprehensive coverage of diverse speech tasks and harness instruction tuning, we invite the community to collaborate and contribute, facilitating the dynamic growth of the benchmark.
Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentation
Because the decoder architecture is the same as an autoregressive LM, it is simple to enhance the model by leveraging external text data with LM training.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction
This pioneering effort aims to set the first benchmark for the AVTSE task, offering fresh insights into enhancing the ac-curacy of back-end speech recognition systems through AVTSE in challenging and real acoustic environments.
Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens
To this end, we start with importing the rich knowledge related to image comprehension and language modeling from a large-scale pre-trained vision-language model into Im2Sp.
Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper
Different from previous methods that tried to improve the VSR performance for the target language by using knowledge learned from other languages, we explore whether we can increase the amount of training data itself for the different languages without human intervention.
Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks
We propose a decoder-only language model, VoxtLM, that can perform four tasks: speech recognition, speech synthesis, text generation, and speech continuation.
Bayes Risk Transducer: Transducer with Controllable Alignment Prediction
While the vanilla transducer does not have a prior preference for any of the valid paths, this work intends to enforce the preferred paths and achieve controllable alignment prediction.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Integration of Frame- and Label-synchronous Beam Search for Streaming Encoder-decoder Speech Recognition
Although frame-based models, such as CTC and transducers, have an affinity for streaming automatic speech recognition, their decoding uses no future knowledge, which could lead to incorrect pruning.
Exploring the Integration of Speech Separation and Recognition with Self-Supervised Learning Representation
In detail, we explore multi-channel separation methods, mask-based beamforming and complex spectral mapping, as well as the best features to use in the ASR back-end model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understanding
There has been an increased interest in the integration of pretrained speech recognition (ASR) and language models (LM) into the SLU framework.
BASS: Block-wise Adaptation for Speech Summarization
End-to-end speech summarization has been shown to improve performance over cascade baselines.
Deep Speech Synthesis from MRI-Based Articulatory Representations
Finally, through a series of ablations, we show that the proposed MRI representation is more comprehensive than EMA and identify the most suitable MRI feature subset for articulatory synthesis.
The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios
The CHiME challenges have played a significant role in the development and evaluation of robust automatic speech recognition (ASR) systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute
Our code and training optimizations make SSL feasible with only 8 GPUs, instead of the 32 used in the original work.
Tensor decomposition for minimization of E2E SLU model toward on-device processing
We reduce the model size by applying tensor decomposition to the Conformer and E-Branchformer architectures used in our E2E SLU models.
DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
Knowledge distillation trains a small student model to mimic the behavior of a large teacher model.
A New Benchmark of Aphasia Speech Recognition and Detection Based on E-Branchformer and Multi-task Learning
Our system achieves state-of-the-art speaker-level detection accuracy (97. 3%), and a relative WER reduction of 11% for moderate Aphasia patients.
Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization
We investigate the emergent abilities of the recently proposed web-scale speech model Whisper, by adapting it to unseen tasks with prompt engineering.
ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks.
A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Tasks
Conformer, a convolution-augmented Transformer variant, has become the de facto encoder architecture for speech processing due to its superior performance in various tasks, including automatic speech recognition (ASR), speech translation (ST) and spoken language understanding (SLU).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation
We use fully unpaired data to train our unsupervised systems and evaluate our results on CoVoST 2 and CVSS.
A Study on the Integration of Pipeline and E2E SLU systems for Spoken Semantic Parsing toward STOP Quality Challenge
Recently there have been efforts to introduce new benchmark tasks for spoken language understanding (SLU), like semantic parsing.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
The Pipeline System of ASR and NLU with MLM-based Data Augmentation toward STOP Low-resource Challenge
In the track, we adopt a pipeline approach of ASR and NLU.
Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog History
Most human interactions occur in the form of spoken conversations where the semantic meaning of a given utterance depends on the context.
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
In this work, we propose a multi-modal AI system named AudioGPT, which complements LLMs (i. e., ChatGPT) with 1) foundation models to process complex audio information and solve numerous understanding and generation tasks; and 2) the input/output interface (ASR, TTS) to support spoken dialogue.
Neural Speech Enhancement with Very Low Algorithmic Latency and Complexity via Integrated Full- and Sub-Band Modeling
We propose FSB-LSTM, a novel long short-term memory (LSTM) based architecture that integrates full- and sub-band (FSB) modeling, for single- and multi-channel speech enhancement in the short-time Fourier transform (STFT) domain.
Efficient Sequence Transduction by Jointly Predicting Tokens and Durations
TDT models for Speech Recognition achieve better accuracy and up to 2. 82X faster inference than conventional Transducers.
Intent Classification
Intent Classification and Slot Filling
+3
ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit
ESPnet-ST-v2 is a revamp of the open-source ESPnet-ST toolkit necessitated by the broadening interests of the spoken language translation community.
Enhancing Speech-to-Speech Translation with Multiple TTS Targets
It has been known that direct speech-to-speech translation (S2ST) models usually suffer from the data scarcity issue because of the limited existing parallel materials for both source and target speech.
I3D: Transformer architectures with input-dependent dynamic depth for speech recognition
Transformer-based end-to-end speech recognition has achieved great success.
End-to-End Speech Recognition: A Survey
In the last decade of automatic speech recognition (ASR) research, the introduction of deep learning brought considerable reductions in word error rate of more than 50% relative, compared to modeling without deep learning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding
Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow.
Improving Massively Multilingual ASR With Auxiliary CTC Objectives
In this paper, we introduce our work on improving performance on FLEURS, a 102-language open ASR benchmark, by conditioning the entire model on language identity (LID).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement
We propose an objective for perceptual quality based on temporal acoustic parameters.
PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech Enhancement
We can add this criterion as an auxiliary loss to any model that produces speech, to optimize speech outputs to match the values of clean speech in these features.
Multi-Channel Target Speaker Extraction with Refinement: The WavLab Submission to the Second Clarity Enhancement Challenge
To address the challenges encountered in the CEC2 setting, we introduce four major novelties: (1) we extend the state-of-the-art TF-GridNet model, originally designed for monaural speaker separation, for multi-channel, causal speech enhancement, and large improvements are observed by replacing the TCNDenseNet used in iNeuBe with this new architecture; (2) we leverage a recent dual window size approach with future-frame prediction to ensure that iNueBe-X satisfies the 5 ms constraint on algorithmic latency required by CEC2; (3) we introduce a novel speaker-conditioning branch for TF-GridNet to achieve target speaker extraction; (4) we propose a fine-tuning step, where we compute an additional loss with respect to the target speaker signal compensated with the listener audiogram.
Speaker-Independent Acoustic-to-Articulatory Speech Inversion
To build speech processing methods that can handle speech as naturally as humans, researchers have explored multiple ways of building an invertible mapping from speech to an interpretable space.
A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech
Recent Text-to-Speech (TTS) systems trained on reading or acted corpora have achieved near human-level naturalness.
Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining
While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data.
Unsupervised Data Selection for TTS: Using Arabic Broadcast News as a Case Study
Several high-resource Text to Speech (TTS) systems currently produce natural, well-established human-like speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders
The network architecture of end-to-end (E2E) automatic speech recognition (ASR) can be classified into several models, including connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), attention mechanism, and non-autoregressive mask-predict models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks
In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape.
Context-aware Fine-tuning of Self-supervised Speech Models
During the fine-tuning stage, we introduce an auxiliary loss that encourages this context embedding vector to be similar to context vectors of surrounding segments.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units
We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization.
SpeechLMScore: Evaluating speech generation using speech language model
While human evaluation is the most reliable metric for evaluating speech generation systems, it is generally costly and time-consuming.
EURO: ESPnet Unsupervised ASR Open-source Toolkit
This paper describes the ESPnet Unsupervised ASR Open-source Toolkit (EURO), an end-to-end open-source toolkit for unsupervised automatic speech recognition (UASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Streaming Joint Speech Recognition and Disfluency Detection
In this study, we propose Transformer-based encoder-decoder models that jointly solve speech recognition and disfluency detection, which work in a streaming manner.
A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units
To address these issues, we devise a cascaded modular system leveraging self-supervised discrete speech units as language representation.
Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generation
To solve this problem, we would like to simultaneously generate automatic speech recognition (ASR) and ST predictions such that each source language word is explicitly mapped to a target language word.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding
Collecting sufficient labeled data for spoken language understanding (SLU) is expensive and time-consuming.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Bridging Speech and Textual Pre-trained Models with Unsupervised ASR
To be specific, we propose to use unsupervised automatic speech recognition (ASR) as a connector that bridges different modalities used in speech and textual pre-trained models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition
In this paper, we propose a new training method to explicitly model and reduce the latency of sequence transducer models.
Multi-blank Transducers for Speech Recognition
This paper proposes a modification to RNN-Transducer (RNN-T) models for automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
InterMPL: Momentum Pseudo-Labeling with Intermediate CTC Loss
This paper presents InterMPL, a semi-supervised learning method of end-to-end automatic speech recognition (ASR) that performs pseudo-labeling (PL) with intermediate supervision.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encoder
One crucial factor that makes this integration challenging lies in the vocabulary mismatch; the vocabulary constructed for a pre-trained LM is generally too large for E2E-ASR training and is likely to have a mismatch against a target ASR domain.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Towards Zero-Shot Code-Switched Speech Recognition
In this work, we seek to build effective code-switched (CS) automatic speech recognition systems (ASR) under the zero-shot setting where no transcribed CS speech data is available for training.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Avoid Overthinking in Self-Supervised Models for Speech Recognition
Although popular for classification tasks in vision and language, EE has seen less use for sequence-to-sequence speech recognition (ASR) tasks where outputs from early layers are often degenerate.
Self-Supervised Learning
Sequence-To-Sequence Speech Recognition
+1
Articulatory Representation Learning Via Joint Factor Analysis and Neural Matrix Factorization
In this work, we propose a novel articulatory representation decomposition algorithm that takes the advantage of guided factor analysis to derive the articulatory-specific factors and factor scores.
BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model
This paper presents BERT-CTC, a novel formulation of end-to-end speech recognition that adapts BERT for connectionist temporal classification (CTC).
Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models
End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation.
In search of strong embedding extractors for speaker diarisation
First, the evaluation is not straightforward because the features required for better performance differ between speaker verification and diarisation.
Large-scale learning of generalised representations for speaker recognition
We also show that training with proposed large data configurations gives better performance.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We present the SUPERB challenge at SLT 2022, which aims at learning self-supervised speech representation for better performance, generalization, and efficiency.
Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks
Besides predicting the target sequence, a side product of CTC is to predict the alignment, which is the most probable input-long sequence that specifies a hard aligning relationship between the input and target units.
On Compressing Sequences for Self-Supervised Speech Models
Subsampling while training self-supervised models not only improves the overall performance on downstream tasks under certain frame rates, but also brings significant speed-up in inference.
CTC Alignments Improve Autoregressive Translation
Connectionist Temporal Classification (CTC) is a widely used approach for automatic speech recognition (ASR) that performs conditionally independent monotonic alignment.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Mutual Learning of Single- and Multi-Channel End-to-End Neural Diarization
This paper focuses on speaker diarization and proposes to conduct the above bi-directional knowledge transfer alternately.
E-Branchformer: Branchformer with Enhanced merging for speech recognition
Conformer, combining convolution and self-attention sequentially to capture both local and global information, has shown remarkable performance and is currently regarded as the state-of-the-art for automatic speech recognition (ASR).
Ranked #9 on
Speech Recognition
on LibriSpeech test-other
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
ESPnet-ONNX: Bridging a Gap Between Research and Production
In the field of deep learning, researchers often focus on inventing novel neural network models and improving benchmarks.
Deep Speech Synthesis from Articulatory Representations
In the articulatory synthesis task, speech is synthesized from input features containing information about the physical behavior of the human vocal tract.
ASR2K: Speech Recognition for Around 2000 Languages without Audio
We achieve 50% CER and 74% WER on the Wilderness dataset with Crubadan statistics only and improve them to 45% CER and 69% WER when using 10000 raw text utterances.
VQ-T: RNN Transducers using Vector-Quantized Prediction Network States
Beam search, which is the dominant ASR decoding algorithm for end-to-end models, generates tree-structured hypotheses.
When Is TTS Augmentation Through a Pivot Language Useful?
Developing Automatic Speech Recognition (ASR) for low-resource languages is a challenge due to the small amount of transcribed audio data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding
To showcase such integration, we performed experiments on carefully designed synthetic datasets for noisy-reverberant multi-channel ST and SLU tasks, which can be used as benchmark corpora for future research.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Two-Pass Low Latency End-to-End Spoken Language Understanding
End-to-end (E2E) models are becoming increasingly popular for spoken language understanding (SLU) systems and are beginning to achieve competitive performance to pipeline-based approaches.
Online Continual Learning of End-to-End Speech Recognition Models
Continual Learning, also known as Lifelong Learning, aims to continually learn from new data as it becomes available.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding
Conformer has proven to be effective in many speech processing tasks.
Improving Speech Enhancement through Fine-Grained Speech Characteristics
We first identify key acoustic parameters that have been found to correlate well with voice quality (e. g. jitter, shimmer, and spectral flux) and then propose objective functions which are aimed at reducing the difference between clean speech and enhanced speech with respect to these features.
Updating Only Encoders Prevents Catastrophic Forgetting of End-to-End ASR Models
In this paper, we present an incremental domain adaptation technique to prevent catastrophic forgetting for an end-to-end automatic speech recognition (ASR) model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Residual Language Model for End-to-end Speech Recognition
In this paper, we propose a simple external LM fusion method for domain adaptation, which considers the internal LM estimation in its training.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
LegoNN: Building Modular Encoder-Decoder Models
We describe LegoNN, a procedure for building encoder-decoder architectures in a way so that its parts can be applied to other tasks without the need for any fine-tuning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors
Finally, to improve online diarization, our method improves the buffer update method and revisits the variable chunk-size training of EEND.
Self-Supervised Speech Representation Learning: A Review
Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages
We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data.
Ranked #3 on
Named Entity Recognition (NER)
on SLUE
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation
Although Transformers have gained success in several speech processing tasks like spoken language understanding (SLU) and speech translation (ST), achieving online processing while keeping competitive performance is still essential for real-world interaction.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation
The present work examines the assumption that combining non-learnable SF extractors to SSL models is an effective approach to low resource speech tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
End-to-End Multi-speaker ASR with Independent Vector Analysis
We develop an end-to-end system for multi-channel, multi-speaker automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representation
This work presents our end-to-end (E2E) automatic speech recognition (ASR) model targetting at robust speech recognition, called Integraded speech Recognition with enhanced speech Input for Self-supervised learning representation (IRIS).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Better Intermediates Improve CTC Inference
This paper proposes a method for improved CTC inference with searched intermediates and multi-pass conditioning.
SingAug: Data Augmentation for Singing Voice Synthesis with Cycle-consistent Training Strategy
Deep learning based singing voice synthesis (SVS) systems have been demonstrated to flexibly generate singing with better qualities, compared to conventional statistical parametric based methods.
EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers
In this paper, we present a novel framework that jointly performs three tasks: speaker diarization, speech separation, and speaker counting.
Memory-Efficient Training of RNN-Transducer with Sampled Softmax
RNN-Transducer has been one of promising architectures for end-to-end automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Investigating self-supervised learning for speech enhancement and separation
Speech enhancement and separation are two fundamental tasks for robust speech processing.
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities
In this paper, we introduce SUPERB-SG, a new benchmark focused on evaluating the semantic and generative capabilities of pre-trained models by increasing task diversity and difficulty over SUPERB.
HEAR: Holistic Evaluation of Audio Representations
The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios.
Extended Graph Temporal Classification for Multi-Speaker End-to-End ASR
As an example application, we use the extended GTC (GTC-e) for the multi-speaker speech recognition task.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Towards Low-distortion Multi-channel Speech Enhancement: The ESPNet-SE Submission to The L3DAS22 Challenge
This paper describes our submission to the L3DAS22 Challenge Task 1, which consists of speech enhancement with 3D Ambisonic microphones.
Acoustic Event Detection with Classifier Chains
In each iteration, the event's activity is estimated and used to condition the next output based on the probabilistic chain rule to form classifier chains.
Conditional Diffusion Probabilistic Model for Speech Enhancement
Speech enhancement is a critical component of many user-oriented audio applications, yet current systems still suffer from distorted and unnatural outputs.
Joint Speech Recognition and Audio Captioning
A major hurdle in evaluating our proposed approach is the lack of labeled audio datasets with both speech transcriptions and audio captions.
Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language models
The proposed NAR model significantly surpasses previous NAR systems on the AISHELL-1 benchmark and shows a potential for English tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Run-and-back stitch search: novel block synchronous decoding for streaming encoder-decoder ASR
To this end, we propose a novel blockwise synchronous decoding algorithm with a hybrid approach that combines endpoint prediction and endpoint post-determination.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies
In this study, we present recent developments of models trained with the RNN-T loss in ESPnet.
Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem
Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols, and convert the paradigm of the speech separation/enhancement related tasks from regression to classification.
ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet
However, there are few open source toolkits that can be used to generate reproducible results on different Spoken Language Understanding (SLU) benchmarks.
Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorization
Conversational bilingual speech encompasses three types of utterances: two purely monolingual types and one intra-sententially code-switched type.
Attention-based Multi-hypothesis Fusion for Speech Summarization
We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Cross-lingual Transfer for Speech Processing using Acoustic Language Similarity
We demonstrate the effectiveness of our approach in language family classification, speech recognition, and speech synthesis tasks.
Sequence Transduction with Graph-based Supervision
The recurrent neural network transducer (RNN-T) objective plays a major role in building today's best automatic speech recognition (ASR) systems for production.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
TorchAudio: Building Blocks for Audio and Speech Processing
This document describes version 0. 10 of TorchAudio: building blocks for machine learning applications in the audio and speech processing domain.
Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions
The deep learning based time-domain models, e. g. Conv-TasNet, have shown great potential in both single-channel and multi-channel speech enhancement.
ESPnet2-TTS: Extending the Edge of TTS Research
This paper describes ESPnet2-TTS, an end-to-end text-to-speech (E2E-TTS) toolkit.
S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations
In this work, we provide a series of in-depth analyses by benchmarking on the two tasks in VCC2020, namely intra-/cross-lingual any-to-one (A2O) VC, as well as an any-to-any (A2A) setting.
A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation
Non-autoregressive (NAR) models simultaneously generate multiple outputs in a sequence, which significantly reduces the inference speed at the cost of accuracy drop compared to autoregressive baselines.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition
The Transformer architecture has been well adopted as a dominant architecture in most sequence transduction tasks including automatic speech recognition (ASR), since its attention mechanism excels in capturing long-range dependencies.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Multi-Channel End-to-End Neural Diarization with Distributed Microphones
With simulated and real-recorded datasets, we demonstrated that the proposed method outperformed conventional EEND when a multi-channel input was given while maintaining comparable performance with a single-channel input.
An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates
We propose Fast-MD, a fast MD model that generates HI by non-autoregressive (NAR) decoding based on connectionist temporal classification (CTC) outputs followed by an ASR decoder.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
We propose a unified NAR E2E-ST framework called Orthros, which has an NAR decoder and an auxiliary shallow AR decoder on top of the shared encoder.
Target-speaker Voice Activity Detection with Improved I-Vector Estimation for Unknown Number of Speaker
Target-speaker voice activity detection (TS-VAD) has recently shown promising results for speaker diarization on highly overlapped speech.
A Study on Speech Enhancement Based on Diffusion Probabilistic Model
Based on this property, we propose a diffusion probabilistic model-based speech enhancement (DiffuSE) model that aims to recover clean speech signals from noisy signals.
Differentiable Allophone Graphs for Language-Universal Speech Recognition
These phone-based systems with learned allophone graphs can be used by linguists to document new languages, build phone-based lexicons that capture rich pronunciation variations, and re-evaluate the allophone mappings of seen language.
On Prosody Modeling for ASR+TTS based Voice Conversion
In voice conversion (VC), an approach showing promising results in the latest voice conversion challenge (VCC) 2020 is to first use an automatic speech recognition (ASR) model to transcribe the source speech into the underlying linguistic contents; these are then used as input by a text-to-speech (TTS) system to generate the converted speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models
Non-autoregressive (NAR) modeling has gained more and more attention in speech processing.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors
This makes it possible to produce diarization results of a large number of speakers for the whole recording even if the number of output speakers for each subsequence is limited.
ESPnet-ST IWSLT 2021 Offline Speech Translation System
This year we made various efforts on training data, architecture, and audio segmentation.
Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding
Our splits identify performance gaps up to 10% between end-to-end systems that were within 1% of each other on the original test sets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Encoder-Decoder Based Attractors for End-to-End Neural Diarization
Diarization results are then estimated as dot products of the attractors and embeddings.
Layer Pruning on Demand with Intermediate CTC
Deploying an end-to-end automatic speech recognition (ASR) model on mobile/embedded devices is a challenging task, since the device computational power and energy consumption requirements are dynamically changed in practice.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Multi-mode Transformer Transducer with Stochastic Future Context
A Multi-mode ASR model can fulfill various latency requirements during inference -- when a larger latency becomes acceptable, the model can process longer future context to achieve higher accuracy and when a latency budget is not flexible, the model can be less dependent on future context but still achieve reliable accuracy.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain
Moreover, by including the data of variable numbers of speakers, our model can even better than the PIT-Conformer AR model with only 1/7 latency, obtaining WERs of 19. 9% and 34. 3% on WSJ0-2mix and WSJ0-3mix sets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio
This paper introduces GigaSpeech, an evolving, multi-domain English speech recognition corpus with 10, 000 hours of high quality labeled audio suitable for supervised training, and 40, 000 hours of total audio suitable for semi-supervised and unsupervised training.
Ranked #1 on
Speech Recognition
on GigaSpeech
Leveraging Pre-trained Language Model for Speech Sentiment Analysis
In this paper, we explore the use of pre-trained language models to learn sentiment information of written texts for speech sentiment analysis.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Semi-Supervised Training with Pseudo-Labeling for End-to-End Neural Diarization
To evaluate our proposed method, we conduct the experiments of model adaptation using labeled and unlabeled data.
End-to-End Speaker Diarization Conditioned on Speech Activity and Overlap Detection
In this paper, we present a conditional multitask learning method for end-to-end neural speaker diarization (EEND).
Data Augmentation Methods for End-to-end Speech Recognition on Distant-Talk Scenarios
Although end-to-end automatic speech recognition (E2E ASR) has achieved great performance in tasks that have numerous paired data, it is still challenging to make E2E ASR robust against noisy and low-resource conditions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
End-to-end ASR to jointly predict transcriptions and linguistic annotations
We propose a Transformer-based sequence-to-sequence model for automatic speech recognition (ASR) capable of simultaneously transcribing and annotating audio with linguistic information such as phonemic transcripts or part-of-speech (POS) tags.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Self-Guided Curriculum Learning for Neural Machine Translation
In the field of machine learning, the well-trained model is assumed to be able to recover the training labels, i. e. the synthetic labels predicted by the model should be as close to the ground-truth labels as possible.
End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings
We present an end-to-end deep network model that performs meeting diarization from single-channel audio recordings.
SUPERB: Speech processing Universal PERformance Benchmark
SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data.
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
In this work, we present an end-to-end framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks.
EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition
Self-supervised ASR-TTS models suffer in out-of-domain data conditions.
Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation
To leverage the full potential of the source language information, we propose backward SeqKD, SeqKD from a target-to-source backward NMT model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition
In the English speech-to-text (STT) machine learning task, acoustic models are conventionally trained on uncased Latin characters, and any necessary orthography (such as capitalization, punctuation, and denormalization of non-standard words) is imputed by separate post-processing models.
Ranked #3 on
Speech Recognition
on SPGISpeech
INTERSPEECH 2021 ConferencingSpeech Challenge: Towards Far-field Multi-Channel Speech Enhancement for Video Conferencing
The ConferencingSpeech 2021 challenge is proposed to stimulate research on far-field multi-channel speech enhancement for video conferencing.
Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yol\'oxochitl Mixtec
{``}Transcription bottlenecks{''}, created by a shortage of effective human transcribers (i. e., transcriber shortage), are one of the main challenges to endangered language (EL) documentation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Dual-Path Modeling for Long Recording Speech Separation in Meetings
A transformer-based dual-path system is proposed, which integrates transform layers for global modeling.
End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend
Recently, the end-to-end approach has been successfully applied to multi-speaker speech separation and recognition in both single-channel and multichannel conditions.
Gaussian Kernelized Self-Attention for Long Sequence Data and Its Application to CTC-based Speech Recognition
Self-attention (SA) based models have recently achieved significant performance improvements in hybrid and end-to-end automatic speech recognition (ASR) systems owing to their flexible context modeling capability.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Deep Learning based Multi-Source Localization with Source Splitting and its Effectiveness in Multi-Talker Speech Recognition
In addition to using the prediction error as a metric for evaluating our localization model, we also establish its potency as a frontend with automatic speech recognition (ASR) as the downstream task.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Intermediate Loss Regularization for CTC-based Speech Recognition
In addition, we propose to combine this intermediate CTC loss with stochastic depth training, and apply this combination to a recently proposed Conformer network.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
The Hitachi-JHU DIHARD III System: Competitive End-to-End Neural Diarization and X-Vector Clustering Systems Combined by DOVER-Lap
This paper provides a detailed description of the Hitachi-JHU system that was submitted to the Third DIHARD Speech Diarization Challenge.
Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yoloxóchitl Mixtec
"Transcription bottlenecks", created by a shortage of effective human transcribers are one of the main challenges to endangered language (EL) documentation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A Review of Speaker Diarization: Recent Advances with Deep Learning
Speaker diarization is a task to label audio or video recordings with classes that correspond to speaker identity, or in short, a task to identify "who spoke when".
Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
As users increasingly rely on cloud-based computing services, it is important to ensure that uploaded speech data remains private.
Online Streaming End-to-End Neural Diarization Handling Overlapping Speech and Flexible Numbers of Speakers
We propose a streaming diarization method based on an end-to-end neural diarization (EEND) model, which handles flexible numbers of speakers and overlapping speech.
Speaker Diarization
Sound
Audio and Speech Processing
Arabic Speech Recognition by End-to-End, Modular Systems and Human
Recent advances in automatic speech recognition (ASR) have achieved accuracy levels comparable to human transcribers, which led researchers to debate if the machine has reached human performance.
