Search Results for author:
Found 76 papers, 40 papers with code
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq Sˆ2, a fairseq extension for speech synthesis.
A Suite for Acoustic Language Model Evaluation
Speech language models have recently demonstrated great potential as universal speech processing systems.
LAST: Language Model Aware Speech Tokenization
Our results demonstrate the proposed tokenization method outperforms the evaluated baselines considering both spoken language modeling and speech-to-text.
Ranked #2 on
Language Modelling
on SALMon
(using extra training data)
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Multi-task Language Understanding
on MMLU
Discrete Flow Matching
Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional discrete data, such as language, is still limited.
A Language Modeling Approach to Diacritic-Free Hebrew TTS
The lack of diacritics in modern Hebrew results in readers expected to conclude the correct pronunciation and understand which phonemes to use based on the context.
HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing
We present HebDB, a weakly supervised dataset for spoken language processing in the Hebrew language.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Improving Visual Commonsense in Language Models via Multiple Image Generation
To facilitate multimodal grounded language modeling, we employ a late-fusion layer that combines the projected visual features with the output of a pre-trained LLM conditioned on text only.
An Independence-promoting Loss for Music Generation with Language Models
This leads to an increase in the generated music quality when modelling the product of the marginal distributions, while generating audio much faster than the joint distribution model.
The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
On the other hand, in scenarios where unit-tests are unavailable, a ranking-based selection of candidates from the smaller model falls short of the performance of a single output from larger ones.
Transformers are Multi-State RNNs
Our results shed light on the connection between transformers and RNNs, and help mitigate one of LLMs' most painful computational bottlenecks - the size of their key-value cache.
Masked Audio Generation using a Single Non-Autoregressive Transformer
We introduce MAGNeT, a masked generative sequence modeling method that operates directly over several streams of audio tokens.
Generative Spoken Language Model based on continuous word-sized audio tokens
In NLP, text language models based on words or subwords are known to outperform their character-based counterparts.
Low-Resource Self-Supervised Learning with SSL-Enhanced TTS
Self-supervised learning (SSL) techniques have achieved remarkable results in various speech processing tasks.
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
The proposed method is based on a lightweight adaptor network, which learns to map the audio-based representation to the input representation expected by the text-to-video generation model.
Code Llama: Open Foundation Models for Code
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks.
Ranked #34 on
Code Generation
on MBPP
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization).
Simple and Controllable Music Generation
We tackle the task of conditional music generation.
Ranked #7 on
Text-to-Music Generation
on MusicCaps
Textually Pretrained Speech Language Models
In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models.
Ranked #4 on
Language Modelling
on SALMon
(using extra training data)
Scaling Speech Technology to 1,000+ Languages
Expanding the language coverage of speech technology has the potential to improve access to information for many more people.
AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
In this paper, we propose a novel method utilizing latent diffusion models trained for text-to-image-generation to generate images conditioned on audio recordings.
Layer Collaboration in the Forward-Forward Algorithm
We show that the current version of the forward-forward algorithm is suboptimal when considering information flow in the network, resulting in a lack of collaboration between layers of the network.
A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation
Expressive speech-to-speech translation (S2ST) aims to transfer prosodic attributes of source speech to target speech while maintaining translation accuracy.
Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling
Following the findings of such an analysis, we propose practical improvements to the discrete unit for the GSLM.
ReVISE: Self-Supervised Speech Resynthesis With Visual Input for Universal and Generalized Speech Regeneration
Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR.
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR.
Ranked #1 on
Speech Recognition
on EasyCom
Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units
We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner.
Ranked #1 on
Voice Conversion
on VCTK
AERO: Audio Super Resolution in the Spectral Domain
We optimize the model using both time and frequency domain loss functions.
Ranked #1 on
Bandwidth Extension
on VCTK
High Fidelity Neural Audio Compression
We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks.
On the Importance of Gradient Norm in PAC-Bayesian Bounds
Generalization bounds which assess the difference between the true risk and the empirical risk, have been studied extensively.
AudioGen: Textually Guided Audio Generation
Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally.
Ranked #13 on
Audio Generation
on AudioCaps
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling
This work focuses on improving the robustness of discrete input representations for generative spoken language modeling.
Deep Audio Waveform Prior
A network with relevant deep priors is likely to generate a cleaner version of the signal before converging on the corrupted signal.
Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors
We proposed an unsupervised method for segmenting symbolic music.
STOP: A dataset for Spoken Task Oriented Semantic Parsing
Furthermore, in addition to the human-recorded audio, we are releasing a TTS-generated version to benchmark the performance for low-resource domain adaptation of end-to-end SLU systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
A Systematic Comparison of Phonetic Aware Techniques for Speech Enhancement
By conducting a series of controlled experiments, we observe the influence of different phonetic content models as well as various feature-injection techniques on enhancement performance, considering both causal and non-causal models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Learning Discrete Structured Variational Auto-Encoder using Natural Evolution Strategies
Discrete variational auto-encoders (VAEs) are able to represent semantic latent spaces in generative learning.
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation
Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues as there exists little parallel S2ST data, compared to the amount of data available for conventional cascaded systems that consist of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) synthesis.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+8
Generative Spoken Dialogue Language Modeling
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues.
Probing phoneme, language and speaker information in unsupervised speech representations
Language information, however, is very salient in the bilingual model only, suggesting CPC models learn to discriminate languages when trained on multiple languages.
RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing
RemixIT is based on a continuous self-training scheme in which a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures.
textless-lib: a Library for Textless Spoken Language Processing
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources.
Textless Speech-to-Speech Translation on Real Data
To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs.
Textless Speech Emotion Conversion using Decomposed & Discrete Representations
We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion.
Textless Speech Emotion Conversion using Discrete and Decomposed Representations
We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion.
Continual self-training with bootstrapped remixing for speech enhancement
Specifically, a separation teacher model is pre-trained on an out-of-domain dataset and is used to infer estimated target signals for a batch of in-domain mixtures.
fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
This paper presents fairseq S^2, a fairseq extension for speech synthesis.
Text-Free Prosody-Aware Generative Spoken Language Modeling
Generative Spoken Language Modeling (GSLM) \cite{Lakhotia2021} is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences.
Ranked #8 on
Language Modelling
on SALMon
(using extra training data)
Direct speech-to-speech translation with discrete units
When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass.
Online Self-Attentive Gated RNNs for Real-Time Speaker Separation
Our stateful implementation for online separation leads to a minor drop in performance compared to the offline model; 0. 8dB for monaural inputs and 0. 3dB for binaural inputs while reaching a real-time factor of 0. 65.
Differentiable Model Compression via Pseudo Quantization Noise
DiffQ is differentiable both with respect to the unquantized weights and the number of bits used.
Ranked #29 on
Language Modelling
on WikiText-103
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
We propose using self-supervised discrete representations for the task of speech resynthesis.
Generative Spoken Language Modeling from Raw Audio
We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation.
Ranked #1 on
Resynthesis
on LibriSpeech
High Fidelity Speech Regeneration with Application to Speech Enhancement
Speech enhancement has seen great improvement in recent years mainly through contributions in denoising, speaker separation, and dereverberation methods that mostly deal with environmental effects on vocal audio.
Single channel voice separation for unknown number of speakers under reverberant and noisy settings
The proposed approach is composed of several separation heads optimized together with a speaker classification branch.
Fairness in the Eyes of the Data: Certifying Machine-Learning Models
We present a framework that allows to certify the fairness degree of a model based on an interactive and privacy-preserving test.
SAGRNN: Self-Attentive Gated RNN for Binaural Speaker Separation with Interaural Cue Preservation
In addition, our approach effectively preserves the interaural cues, which improves the accuracy of sound localization.
Audio and Speech Processing Sound
Unsupervised Cross-Domain Singing Voice Conversion
We present a wav-to-wav generative model for the task of singing voice conversion from any identity.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation
Results suggest that our approach surpasses the baseline models and reaches state-of-the-art performance on both data sets.
Real Time Speech Enhancement in the Waveform Domain
The proposed model matches state-of-the-art performance of both causal and non causal methods while working directly on the raw waveform.
Voice Separation with an Unknown Number of Multiple Speakers
We present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously.
Ranked #2 on
Speech Separation
on WSJ0-4mix
On the generalization of bayesian deep nets for multi-class classification
Generalization bounds which assess the difference between the true risk and the empirical risk have been studied extensively.
Phoneme Boundary Detection using Learnable Segmental Features
Phoneme boundary detection plays an essential first step for a variety of speech processing applications such as speaker diarization, speech science, keyword spotting, etc.
PAC-Bayesian Neural Network Bounds
Bayesian neural networks, which both use the negative log-likelihood loss function and average their predictions using a learned posterior over the parameters, have been used successfully across many scientific fields, partly due to their ability to `effortlessly' extract desired representations from many large-scale datasets.
Hide and Speak: Towards Deep Neural Networks for Speech Steganography
Steganography is the science of hiding a secret message within an ordinary public message, which is referred to as Carrier.
To Reverse the Gradient or Not: An Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition
In multi-task learning, the goal is speaker prediction; we expect a performance improvement with this joint training if the two tasks of speech recognition and speaker recognition share a common set of underlying features.
Out-of-Distribution Detection using Multiple Semantic Label Representations
Deep Neural Networks are powerful models that attained remarkable results on a variety of tasks.
Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring
Unfortunately, once the models are sold they can be easily copied and redistributed.
Fooling End-to-end Speaker Verification by Adversarial Examples
We also present two black-box attacks: where the adversarial examples were generated with a system that was trained on YOHO, but the attack is on a system that was trained on NTIMIT; and when the adversarial examples were generated with a system that was trained on Mel-spectrum feature set, but the attack is on a system that was trained on MFCC.
Houdini: Fooling Deep Structured Visual and Speech Recognition Models with Adversarial Examples
Generating adversarial examples is a critical step for evaluating and improving the robustness of learning machines.
Houdini: Fooling Deep Structured Prediction Models
Generating adversarial examples is a critical step for evaluating and improving the robustness of learning machines.
Automatic Measurement of Pre-aspiration
Pre-aspiration is defined as the period of glottal friction occurring in sequences of vocalic/consonantal sonorants and phonetically voiceless obstruents.
Learning Similarity Functions for Pronunciation Variations
This task generalizes problems such as lexical access (the problem of learning the mapping between words and their possible pronunciations), and defining word neighborhoods.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Automatic measurement of vowel duration via structured prediction
Manually-annotated data were used to train a model that takes as input an arbitrary length segment of the acoustic signal containing a single vowel that is preceded and followed by consonants and outputs the duration of the vowel.
Sequence Segmentation Using Joint RNN and Structured Prediction Models
We describe and analyze a simple and effective algorithm for sequence segmentation applied to speech processing tasks.
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
The analysis sheds light on the relative strengths of different sentence embedding methods with respect to these low level prediction tasks, and on the effect of the encoded vector's dimensionality on the resulting representations.
