Search Results for author:
Found 64 papers, 32 papers with code
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq Sˆ2, a fairseq extension for speech synthesis.
Code Llama: Open Foundation Models for Code
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks.
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization).
From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion
Recently, such models have been used to synthesize audio waveforms conditioned on highly compressed representations.
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale.
Simple and Controllable Music Generation
We tackle the task of conditional music generation.
Ranked #4 on
Text-to-Music Generation
on MusicCaps
AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
In this paper, we propose a novel method utilizing latent diffusion models trained for text-to-image-generation to generate images conditioned on audio recordings.
Scaling Speech Technology to 1,000+ Languages
Expanding the language coverage of speech technology has the potential to improve access to information for many more people.
Textually Pretrained Speech Language Models
Based on our observations, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data.
Layer Collaboration in the Forward-Forward Algorithm
We show that the current version of the forward-forward algorithm is suboptimal when considering information flow in the network, resulting in a lack of collaboration between layers of the network.
A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation
Expressive speech-to-speech translation (S2ST) aims to transfer prosodic attributes of source speech to target speech while maintaining translation accuracy.
Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling
Following the findings of such an analysis, we propose practical improvements to the discrete unit for the GSLM.
ReVISE: Self-Supervised Speech Resynthesis With Visual Input for Universal and Generalized Speech Regeneration
Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR.
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR.
Ranked #1 on
Speech Recognition
on EasyCom
Speaking Style Conversion With Discrete Self-Supervised Units
We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate the proposed approach is significantly superior to the evaluated baselines.
AERO: Audio Super Resolution in the Spectral Domain
We optimize the model using both time and frequency domain loss functions.
Ranked #1 on
Bandwidth Extension
on VCTK
High Fidelity Neural Audio Compression
We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks.
On the Importance of Gradient Norm in PAC-Bayesian Bounds
Generalization bounds which assess the difference between the true risk and the empirical risk, have been studied extensively.
AudioGen: Textually Guided Audio Generation
Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally.
Ranked #8 on
Audio Generation
on AudioCaps
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling
This work focuses on improving the robustness of discrete input representations for generative spoken language modeling.
Deep Audio Waveform Prior
A network with relevant deep priors is likely to generate a cleaner version of the signal before converging on the corrupted signal.
Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors
We proposed an unsupervised method for segmenting symbolic music.
STOP: A dataset for Spoken Task Oriented Semantic Parsing
Furthermore, in addition to the human-recorded audio, we are releasing a TTS-generated version to benchmark the performance for low-resource domain adaptation of end-to-end SLU systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
A Systematic Comparison of Phonetic Aware Techniques for Speech Enhancement
By conducting a series of controlled experiments, we observe the influence of different phonetic content models as well as various feature-injection techniques on enhancement performance, considering both causal and non-causal models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Learning Discrete Structured Variational Auto-Encoder using Natural Evolution Strategies
Discrete variational auto-encoders (VAEs) are able to represent semantic latent spaces in generative learning.
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation
Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues as there exists little parallel S2ST data, compared to the amount of data available for conventional cascaded systems that consist of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) synthesis.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Probing phoneme, language and speaker information in unsupervised speech representations
Language information, however, is very salient in the bilingual model only, suggesting CPC models learn to discriminate languages when trained on multiple languages.
Generative Spoken Dialogue Language Modeling
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues.
RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing
RemixIT is based on a continuous self-training scheme in which a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures.
textless-lib: a Library for Textless Spoken Language Processing
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources.
Textless Speech-to-Speech Translation on Real Data
To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs.
Textless Speech Emotion Conversion using Decomposed & Discrete Representations
We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion.
Textless Speech Emotion Conversion using Discrete and Decomposed Representations
We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion.
Continual self-training with bootstrapped remixing for speech enhancement
Specifically, a separation teacher model is pre-trained on an out-of-domain dataset and is used to infer estimated target signals for a batch of in-domain mixtures.
fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq S^2, a fairseq extension for speech synthesis.
Text-Free Prosody-Aware Generative Spoken Language Modeling
Generative Spoken Language Modeling (GSLM) \cite{Lakhotia2021} is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences.
Direct speech-to-speech translation with discrete units
When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass.
Online Self-Attentive Gated RNNs for Real-Time Speaker Separation
Our stateful implementation for online separation leads to a minor drop in performance compared to the offline model; 0. 8dB for monaural inputs and 0. 3dB for binaural inputs while reaching a real-time factor of 0. 65.
Differentiable Model Compression via Pseudo Quantization Noise
DiffQ is differentiable both with respect to the unquantized weights and the number of bits used.
Ranked #25 on
Language Modelling
on WikiText-103
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
We propose using self-supervised discrete representations for the task of speech resynthesis.
Generative Spoken Language Modeling from Raw Audio
We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation.
Ranked #1 on
Resynthesis
on LJSpeech
High Fidelity Speech Regeneration with Application to Speech Enhancement
Speech enhancement has seen great improvement in recent years mainly through contributions in denoising, speaker separation, and dereverberation methods that mostly deal with environmental effects on vocal audio.
Single channel voice separation for unknown number of speakers under reverberant and noisy settings
The proposed approach is composed of several separation heads optimized together with a speaker classification branch.
Fairness in the Eyes of the Data: Certifying Machine-Learning Models
We present a framework that allows to certify the fairness degree of a model based on an interactive and privacy-preserving test.
SAGRNN: Self-Attentive Gated RNN for Binaural Speaker Separation with Interaural Cue Preservation
In addition, our approach effectively preserves the interaural cues, which improves the accuracy of sound localization.
Audio and Speech Processing Sound
Unsupervised Cross-Domain Singing Voice Conversion
We present a wav-to-wav generative model for the task of singing voice conversion from any identity.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation
Results suggest that our approach surpasses the baseline models and reaches state-of-the-art performance on both data sets.
Real Time Speech Enhancement in the Waveform Domain
The proposed model matches state-of-the-art performance of both causal and non causal methods while working directly on the raw waveform.
Voice Separation with an Unknown Number of Multiple Speakers
We present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously.
Ranked #1 on
Speech Separation
on WSJ0-4mix
On the generalization of bayesian deep nets for multi-class classification
Generalization bounds which assess the difference between the true risk and the empirical risk have been studied extensively.
Phoneme Boundary Detection using Learnable Segmental Features
Phoneme boundary detection plays an essential first step for a variety of speech processing applications such as speaker diarization, speech science, keyword spotting, etc.
PAC-Bayesian Neural Network Bounds
Bayesian neural networks, which both use the negative log-likelihood loss function and average their predictions using a learned posterior over the parameters, have been used successfully across many scientific fields, partly due to their ability to `effortlessly' extract desired representations from many large-scale datasets.
Hide and Speak: Towards Deep Neural Networks for Speech Steganography
Steganography is the science of hiding a secret message within an ordinary public message, which is referred to as Carrier.
To Reverse the Gradient or Not: An Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition
In multi-task learning, the goal is speaker prediction; we expect a performance improvement with this joint training if the two tasks of speech recognition and speaker recognition share a common set of underlying features.
Out-of-Distribution Detection using Multiple Semantic Label Representations
Deep Neural Networks are powerful models that attained remarkable results on a variety of tasks.
Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring
Unfortunately, once the models are sold they can be easily copied and redistributed.
Fooling End-to-end Speaker Verification by Adversarial Examples
We also present two black-box attacks: where the adversarial examples were generated with a system that was trained on YOHO, but the attack is on a system that was trained on NTIMIT; and when the adversarial examples were generated with a system that was trained on Mel-spectrum feature set, but the attack is on a system that was trained on MFCC.
Houdini: Fooling Deep Structured Visual and Speech Recognition Models with Adversarial Examples
Generating adversarial examples is a critical step for evaluating and improving the robustness of learning machines.
Houdini: Fooling Deep Structured Prediction Models
Generating adversarial examples is a critical step for evaluating and improving the robustness of learning machines.
Automatic Measurement of Pre-aspiration
Pre-aspiration is defined as the period of glottal friction occurring in sequences of vocalic/consonantal sonorants and phonetically voiceless obstruents.
Learning Similarity Functions for Pronunciation Variations
This task generalizes problems such as lexical access (the problem of learning the mapping between words and their possible pronunciations), and defining word neighborhoods.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Automatic measurement of vowel duration via structured prediction
Manually-annotated data were used to train a model that takes as input an arbitrary length segment of the acoustic signal containing a single vowel that is preceded and followed by consonants and outputs the duration of the vowel.
Sequence Segmentation Using Joint RNN and Structured Prediction Models
We describe and analyze a simple and effective algorithm for sequence segmentation applied to speech processing tasks.
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
The analysis sheds light on the relative strengths of different sentence embedding methods with respect to these low level prediction tasks, and on the effect of the encoded vector's dimensionality on the resulting representations.
