Search Results for author:
Found 95 papers, 34 papers with code
Training and Inference Efficiency of Encoder-Decoder Speech Models
However, the reported data and compute requirements for their training are prohibitive for many in the research community.
Scoring Verifiers: Evaluating Synthetic Verification in Code and Reasoning
Building on these advancements, we propose new benchmarks designed to systematically evaluate the impact of synthetic verification methods on assessing solution correctness.
Methods to Increase the Amount of Data for Speech Recognition for Low Resource Languages
This study explores methods to increase data volume for low-resource languages using techniques such as crowdsourcing, pseudo-labeling, advanced data preprocessing and various permissive data sources such as audiobooks, Common Voice, YouTube.
Star Attention: Efficient LLM Inference over Long Sequences
Inference with Transformer-based Large Language Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism.
NeKo: Toward Post Recognition Generative Correction Large Language Models with Task-Oriented Experts
On zero-shot evaluation, NeKo outperforms GPT-3. 5 and Claude-Opus with $15. 5$% to $27. 6$% relative WER reduction in the Hyporadise benchmark.
Anticipating Future with Large Language Model for Simultaneous Machine Translation
Simultaneous machine translation (SMT) takes streaming input utterances and incrementally produces target text.
VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning
More recent studies have extended this to multi-turn conversations, though they often require complex, multi-stage supervised fine-tuning (SFT) with diverse data.
Three-in-One: Fast and Accurate Transducer for Hybrid-Autoregressive ASR
We present \textbf{H}ybrid-\textbf{A}utoregressive \textbf{IN}ference Tr\textbf{AN}sducers (HAINAN), a novel architecture for speech recognition that extends the Token-and-Duration Transducer (TDT) model.
nGPT: Normalized Transformer with Representation Learning on the Hypersphere
We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere.
DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data
Recent end-to-end speech language models (SLMs) have expanded upon the capabilities of large language models (LLMs) by incorporating pre-trained speech models.
EMMeTT: Efficient Multimodal Machine Translation Training
This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST).
META-CAT: Speaker-Informed Speech Embeddings via Meta Information Concatenation for Multi-talker ASR
We propose a novel end-to-end multi-talker automatic speech recognition (ASR) framework that enables both multi-speaker (MS) ASR and target-speaker (TS) ASR.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Chain-of-Thought Prompting for Speech Translation
Building on the success of text-based LLMs, recent research has adapted these models to use speech embeddings for prompting, resulting in Speech-LLM models that exhibit strong performance in automatic speech recognition (ASR) and automatic speech translation (AST).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition
Given recent advances in generative AI technology, a key question is how large language models (LLMs) can enhance acoustic modeling tasks using text decoding results from a frozen, pretrained automatic speech recognition (ASR) model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Sortformer: Seamless Integration of Speaker Diarization and ASR by Bridging Timestamps and Tokens
We demonstrate that combining Sort Loss and PIL achieves performance competitive with state-of-the-art end-to-end diarization models trained exclusively with PIL.
Longer is (Not Necessarily) Stronger: Punctuated Long-Sequence Training for Enhanced Speech Recognition and Translation
Additionally, training on longer audio segments increases the overall model accuracy across speech recognition and translation benchmarks.
Resource-Efficient Adaptation of Speech Foundation Models for Multi-Speaker ASR
Speech foundation models have achieved state-of-the-art (SoTA) performance across various tasks, such as automatic speech recognition (ASR) in hundreds of languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models
Large Language Models (LLMs) rely on instruction samples for alignment, but creating these datasets poses challenges, particularly in expert-dependent tasks like coding, which can be cost-prohibitive.
Schrödinger Bridge for Generative Speech Enhancement
This paper proposes a generative speech enhancement model based on Schr\"odinger bridge (SB).
Ranked #4 on
Speech Enhancement
on EARS-WHAM
Romanization Encoding For Multilingual ASR
Ablation studies on Mandarin-Korean and Mandarin-Japanese highlight our method's strong capability to address the complexities of other script-heavy languages, paving the way for more versatile and effective multilingual ASR systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations
Discrete speech representations have garnered recent attention for their efficacy in training transformer-based models for various speech-related tasks such as automatic speech recognition (ASR), translation, speaker verification, and joint speech-text foundational models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5
We propose BESTOW architecture to bring the BESt features from TwO Worlds into a single model that is highly efficient and has strong multitask capabilities.
Less is More: Accurate Speech Recognition & Translation without Web-Scale Data
Recent advances in speech recognition and translation rely on hundreds of thousands of hours of Internet speech data.
DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language models (LLMs).
Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment
Large Language Model (LLM) based text-to-speech (TTS) systems have demonstrated remarkable capabilities in handling large speech datasets and generating natural speech for new speakers.
Instruction Data Generation and Unsupervised Adaptation for Speech Language Models
In this paper, we propose three methods for generating synthetic samples to train and evaluate multimodal large language models capable of processing both text and speech inputs.
Nemotron-4 340B Technical Report
We release the Nemotron-4 340B model family, including Nemotron-4-340B-Base, Nemotron-4-340B-Instruct, and Nemotron-4-340B-Reward.
Fast Context-Biasing for CTC and Transducer ASR models with CTC-based Word Spotter
Accurate recognition of rare and new words remains a pressing problem for contextualized Automatic Speech Recognition (ASR) systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Label-Looping: Highly Efficient Decoding for Transducers
This paper introduces a highly efficient greedy decoding algorithm for Transducer-based speech recognition models.
Spectral Codecs: Spectrogram-Based Audio Codecs for High Quality Speech Synthesis
Recently, discrete audio tokens produced by neural audio codecs have become a popular alternate speech representation for speech synthesis tasks such as text-to-speech (TTS).
Flexible Multichannel Speech Enhancement for Noise-Robust Frontend
This paper proposes a flexible multichannel speech enhancement system with the main goal of improving robustness of automatic speech recognition (ASR) in noisy conditions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
RULER: What's the Real Context Size of Your Long-Context Language Models?
Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases.
Transducers with Pronunciation-aware Embeddings for Automatic Speech Recognition
This paper proposes Transducers with Pronunciation-aware Embeddings (PET).
Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer
Humans are adept at leveraging visual cues from lip movements for recognizing speech in adverse listening conditions.
Audio-Visual Speech Recognition
Robust Speech Recognition
+2
Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition
We also showed that training a model with multiple latencies can achieve better accuracy than single latency models while it enables us to support multiple latencies with a single model.
Property-Aware Multi-Speaker Data Simulation: A Probabilistic Modelling Technique for Synthetic Data Generation
This capability offers a tailored training environment for developing neural models suited for speaker diarization and voice activity detection.
The CHiME-7 Challenge: System Description and Performance of NeMo Team's DASR System
We present the NVIDIA NeMo team's multi-channel speech recognition system for the 7th CHiME Challenge Distant Automatic Speech Recognition (DASR) Task, focusing on the development of a multi-channel, multi-speaker speech recognition system tailored to transcribe speech from distributed microphones and microphone arrays.
SelfVC: Voice Conversion With Iterative Refinement using Self Transformations
In this work, instead of explicitly disentangling attributes with loss terms, we present a framework to train a controllable voice conversion model on entangled speech representations derived from self-supervised learning (SSL) and speaker verification models.
SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation
We present a novel Speech Augmented Language Model (SALM) with {\em multitask} and {\em in-context} learning capabilities.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models
Traditional automatic speech recognition (ASR) models output lower-cased words without punctuation marks, which reduces readability and necessitates a subsequent text processing model to convert ASR transcripts into a proper format.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A Chat About Boring Problems: Studying GPT-based text normalization
Through this new framework, we can identify strengths and weaknesses of GPT-based TN, opening opportunities for future work.
Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition
Discrete audio representation, aka audio tokenization, has seen renewed interest driven by its potential to facilitate the application of text language modeling approaches in audio domain.
Investigating End-to-End ASR Architectures for Long Form Audio Transcription
This paper presents an overview and evaluation of some of the end-to-end ASR models on long-form audios.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audio
We propose CONF-TSASR, a non-autoregressive end-to-end time-frequency domain architecture for single-channel target-speaker automatic speech recognition (TS-ASR).
Leveraging Pretrained ASR Encoders for Effective and Efficient End-to-End Speech Intent Classification and Slot Filling
We study speech intent classification and slot filling (SICSF) by proposing to use an encoder pretrained on speech recognition (ASR) to initialize an end-to-end (E2E) Conformer-Transformer model, which achieves the new state-of-the-art results on the SLURP dataset, with 90. 14% intent accuracy and 82. 27% SLURP-F1.
Confidence-based Ensembles of End-to-End Speech Recognition Models
Second, we demonstrate that it is possible to combine base and adapted models to achieve strong results on both original and target data.
Unified model for code-switching speech recognition and language identification based on a concatenated tokenizer
Code-Switching (CS) multilingual Automatic Speech Recognition (ASR) models can transcribe speech containing two or more alternating languages during a conversation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
SpellMapper: A non-autoregressive neural spellchecker for ASR customization with candidate retrieval based on n-gram mappings
Contextual spelling correction models are an alternative to shallow fusion to improve automatic speech recognition (ASR) quality given user vocabulary.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
Conformer-based models have become the dominant end-to-end architecture for speech processing tasks.
Ranked #1 on
Speech Recognition
on Common Voice English
(using extra training data)
Efficient Sequence Transduction by Jointly Predicting Tokens and Durations
TDT models for Speech Recognition achieve better accuracy and up to 2. 82X faster inference than conventional Transducers.
Intent Classification
Intent Classification and Slot Filling
+3
Powerful and Extensible WFST Framework for RNN-Transducer Losses
This paper presents a framework based on Weighted Finite-State Transducers (WFST) to simplify the development of modifications for RNN-Transducer (RNN-T) loss.
VANI: Very-lightweight Accent-controllable TTS for Native and Non-native speakers with Identity Preservation
We introduce VANI, a very lightweight multi-lingual accent controllable speech synthesis system.
Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator
We propose an end-to-end Automatic Speech Recognition (ASR) system that can be trained on transcribed speech data, text-only data, or a mixture of both.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations
In this work, we propose a zero-shot voice conversion method using speech representations trained with self-supervised learning.
Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognition
This paper presents a class of new fast non-trainable entropy-based confidence estimation methods for automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models
In this paper, we extend previous self-supervised approaches for language identification by experimenting with Conformer based architecture in a multilingual pre-training paradigm.
Multi-blank Transducers for Speech Recognition
This paper proposes a modification to RNN-Transducer (RNN-T) models for automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New Speakers
In the proposed approach, a few small adapter modules are added to the original network.
A Compact End-to-End Model with Local and Global Context for Spoken Language Identification
We introduce TitaNet-LID, a compact end-to-end neural network for Spoken Language Identification (LID) that is based on the ContextNet architecture.
Damage Control During Domain Adaptation for Transducer Based Automatic Speech Recognition
Automatic speech recognition models are often adapted to improve their accuracy in a new domain.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Thutmose Tagger: Single-pass neural model for Inverse Text Normalization
The model is trained on the Google Text Normalization dataset and achieves state-of-the-art sentence accuracy on both English and Russian test sets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
BigVGAN: A Universal Neural Vocoder with Large-Scale Training
Despite recent progress in generative adversarial network (GAN)-based vocoders, where the model generates raw waveform conditioned on acoustic features, it is challenging to synthesize high-fidelity audio for numerous speakers across various recording environments.
Ranked #2 on
Speech Synthesis
on LibriTTS
(using extra training data)
Multi-scale Speaker Diarization with Dynamic Scale Weighting
First, we use multi-scale clustering as an initialization to estimate the number of speakers and obtain the average speaker representation vector for each speaker and each scale.
Shallow Fusion of Weighted Finite-State Transducer and Language Model for Text Normalization
First, a non-deterministic WFST outputs all normalization candidates, and then a neural language model picks the best one -- similar to shallow fusion for automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Adapting TTS models For New Speakers using Transfer Learning
We address this challenge by proposing transfer-learning guidelines for adapting high quality single-speaker TTS models for a new speaker, using only a few minutes of speech data.
TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context
In this paper, we propose TitaNet, a novel neural network architecture for extracting speaker representations.
Ranked #1 on
Speaker Diarization
on CALLHOME-109
Mixer-TTS: non-autoregressive, fast and compact text-to-speech model conditioned on language model embeddings
This paper describes Mixer-TTS, a non-autoregressive model for mel-spectrogram generation.
CTC Variations Through New WFST Topologies
This paper presents novel Weighted Finite-State Transducer (WFST) topologies to implement Connectionist Temporal Classification (CTC)-like algorithms for automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A Unified Transformer-based Framework for Duplex Text Normalization
In addition, we also create a cleaned dataset from the Spoken Wikipedia Corpora for German and report the performance of our systems on the dataset.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
CarneliNet: Neural Mixture Model for Automatic Speech Recognition
The basic idea is to introduce a parallel mixture of shallow networks instead of a very deep network.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Services
In this paper, we propose SGD-QA, a simple and extensible model for schema-guided dialogue state tracking based on a question answering approach.
TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Prediction
We propose TalkNet, a non-autoregressive convolutional neural model for speech synthesis with explicit pitch and duration prediction.
NeMo Inverse Text Normalization: From Development To Production
Inverse text normalization (ITN) converts spoken-domain automatic speech recognition (ASR) output into written-domain text to improve the readability of the ASR output.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A Toolbox for Construction and Analysis of Speech Datasets
Automatic Speech Recognition and Text-to-Speech systems are primarily trained in a supervised fashion and require high-quality, accurately labeled speech datasets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition
In the English speech-to-text (STT) machine learning task, acoustic models are conventionally trained on uncased Latin characters, and any necessary orthography (such as capitalization, punctuation, and denormalization of non-standard words) is imputed by separate post-processing models.
Ranked #3 on
Speech Recognition
on SPGISpeech
Citrinet: Closing the Gap between Non-Autoregressive and Autoregressive End-to-End Models for Automatic Speech Recognition
We propose Citrinet - a new end-to-end convolutional Connectionist Temporal Classification (CTC) based automatic speech recognition (ASR) model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Hi-Fi Multi-Speaker English TTS Dataset
This paper introduces a new multi-speaker English dataset for training text-to-speech models.
On regularization of gradient descent, layer imbalance and flat minima
We analyze the training dynamics for deep linear networks using a new metric - layer imbalance - which defines the flatness of a solution.
Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay.
Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model
In this work, we introduce a simple yet efficient post-processing model for automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions
We propose a new end-to-end neural acoustic model for automatic speech recognition.
Ranked #41 on
Speech Recognition
on LibriSpeech test-other
Speech Recognition
Audio and Speech Processing
NeMo: a toolkit for building AI applications using Neural Modules
NeMo (Neural Modules) is a Python framework-agnostic toolkit for creating AI applications through re-usability, abstraction, and composition.
Ranked #1 on
Speech Recognition
on Common Voice Spanish
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay.
Jasper: An End-to-End Convolutional Neural Acoustic Model
In this paper, we report state-of-the-art results on LibriSpeech among end-to-end speech recognition models without any external training data.
Ranked #3 on
Speech Recognition
on Hub5'00 SwitchBoard
Training Neural Speech Recognition Systems with Synthetic Speech Augmentation
Building an accurate automatic speech recognition (ASR) system requires a large dataset that contains many hours of labeled speech samples produced by a diverse set of speakers.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
OpenSeq2Seq: Extensible Toolkit for Distributed and Mixed Precision Training of Sequence-to-Sequence Models
We present OpenSeq2Seq {--} an open-source toolkit for training sequence-to-sequence models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
We present OpenSeq2Seq - a TensorFlow-based toolkit for training sequence-to-sequence models that features distributed and mixed-precision training.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Training Deep AutoEncoders for Recommender Systems
Our model is based on deep autoencoder with 6 layers and is trained end-to-end without any layer-wise pre-training.
Large Batch Training of Convolutional Networks with Layer-wise Adaptive Rate Scaling
Using LARS, we scaled AlexNet and ResNet-50 to a batch size of 16K.
Mixed Precision Training
Using this approach, we can reduce the memory consumption of deep learning models by nearly 2x.
Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
However, it is not clear if these algorithms could replace BN in practical, large-scale applications.
Large Batch Training of Convolutional Networks
Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.
Training Deep AutoEncoders for Collaborative Filtering
Our model is based on deep autoencoder with 6 layers and is trained end-to-end without any layer-wise pre-training.
Factorization tricks for LSTM networks
We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups.
Ranked #22 on
Language Modelling
on One Billion Word
SEBOOST - Boosting Stochastic Learning Using Subspace Optimization Techniques
SEBOOST applies a secondary optimization process in the subspace spanned by the last steps and descent directions.
