Search Results for author:
Found 20 papers, 10 papers with code
TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages
We propose a novel Tri-Modal Translation (TMT) model that translates between arbitrary modalities spanning speech, image, and text.
SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition
Existing speech language models typically utilize task-dependent prompt tokens to unify various speech tasks in a single model.
Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech
In this paper, we propose an evaluation technique involving the training of an ASR model on synthetic speech and assessing its performance on real speech.
Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling.
Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learning
We show that further efficiency can be achieved with a vanilla HuBERT Base model, which can maintain 94% of XLS-R's performance with only 3% of the data, 4 GPUs, and limited trials.
Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
Pre-training speech models on large volumes of data has achieved remarkable success.
Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens
To this end, we start with importing the rich knowledge related to image comprehension and language modeling from a large-scale pre-trained vision-language model into Im2Sp.
Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks
We propose a decoder-only language model, VoxtLM, that can perform four tasks: speech recognition, speech synthesis, text generation, and speech continuation.
Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute
Our code and training optimizations make SSL feasible with only 8 GPUs, instead of the 32 used in the original work.
ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit
ESPnet-ST-v2 is a revamp of the open-source ESPnet-ST toolkit necessitated by the broadening interests of the spoken language translation community.
Improving Massively Multilingual ASR With Auxiliary CTC Objectives
In this paper, we introduce our work on improving performance on FLEURS, a 102-language open ASR benchmark, by conditioning the entire model on language identity (LID).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining
While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data.
Unsupervised Data Selection for TTS: Using Arabic Broadcast News as a Case Study
Several high-resource Text to Speech (TTS) systems currently produce natural, well-established human-like speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
SpeechLMScore: Evaluating speech generation using speech language model
While human evaluation is the most reliable metric for evaluating speech generation systems, it is generally costly and time-consuming.
EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers
In this paper, we present a novel framework that jointly performs three tasks: speaker diarization, speech separation, and speaker counting.
End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings
We present an end-to-end deep network model that performs meeting diarization from single-channel audio recordings.
Generating Multilingual Voices Using Speaker Space Translation Based on Bilingual Speaker Data
We demonstrate that a bilingual speaker embedding space contains a separate distribution for each language and that a simple transform in speaker space generated by the speaker embedding can be used to control the degree of accent of a synthetic voice in a language.
Speaker independence of neural vocoders and their effect on parametric resynthesis speech enhancement
In previous work, we showed that PR systems generate high quality speech for a single speaker using two neural vocoders, WaveNet and WaveGlow.
Parametric Resynthesis with neural vocoders
We propose to utilize the high quality speech generation capability of neural vocoders for noise suppression.
Speech denoising by parametric resynthesis
In comparison to two denoising systems, the oracle Wiener mask and a DNN-based mask predictor, our model equals the oracle Wiener mask in subjective quality and intelligibility and surpasses the realistic system.
