Search Results for author:
Found 35 papers, 15 papers with code
MAD Speech: Measures of Acoustic Diversity of Speech
Using these datasets, we demonstrate that our proposed metrics achieve a stronger agreement with the ground-truth diversity than baselines.
MusicRL: Aligning Music Generation to Human Preferences
MusicRL is a pretrained autoregressive MusicLM (Agostinelli et al., 2023) model of discrete audio tokens finetuned with reinforcement learning to maximise sequence-level rewards.
TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition
The model operates on transcripts and audio token sequences and achieves multiple tasks through masking of inputs.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
AudioPaLM: A Large Language Model That Can Speak and Listen
AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2.
SoundStorm: Efficient Parallel Audio Generation
We present SoundStorm, a model for efficient, non-autoregressive audio generation.
Speech Intelligibility Classifiers from 550k Disordered Speech Samples
We developed dysarthric speech intelligibility classifiers on 551, 176 disordered speech samples contributed by a diverse set of 468 speakers, with a range of self-reported speaking disorders and rated for their overall intelligibility on a five-point scale.
DNArch: Learning Convolutional Neural Architectures by Backpropagation
We present Differentiable Neural Architectures (DNArch), a method that jointly learns the weights and the architecture of Convolutional Neural Networks (CNNs) by backpropagation.
SingSong: Generating musical accompaniments from singing
We present SingSong, a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create music featuring their own voice.
MusicLM: Generating Music From Text
We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff".
Ranked #8 on
Text-to-Music Generation
on MusicCaps
AudioLM: a Language Modeling Approach to Audio Generation
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency.
Multi-instrument Music Synthesis with Spectrogram Diffusion
An ideal music synthesizer should be both interactive and expressive, generating high-fidelity audio in realtime for arbitrary combinations of instruments and notes.
Learning neural audio features without supervision
Deep audio classification, traditionally cast as training a deep neural network on top of mel-filterbanks in a supervised fashion, has recently benefited from two independent lines of work.
Disentangling speech from surroundings with neural embeddings
We present a method to separate speech signals from noisy environments in the embedding space of a neural audio codec.
General-purpose, long-context autoregressive modeling with Perceiver AR
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression.
Ranked #35 on
Language Modelling
on WikiText-103
Learning strides in convolutional neural networks
In particular, we show that introducing our layer into a ResNet-18 architecture allows keeping consistent high performance on CIFAR10, CIFAR100 and ImageNet even when training starts from poor random stride configurations.
SoundStream: An End-to-End Neural Audio Codec
We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs.
DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding
Our neural algorithm presents the diarization task as an iterative process: it repeatedly builds a representation for each speaker before predicting the voice activity of each speaker conditioned on the extracted representations.
Self-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking
Second, we show that inverting permutations is a meaningful pretext task for learning audio representations in an unsupervised fashion.
LEAF: A Learnable Frontend for Audio Classification
In this work we show that we can train a single learnable frontend that outperforms mel-filterbanks on a wide range of audio signals, including speech, music, audio events and animal sounds, providing a general-purpose learned frontend for audio classification.
Shuffle to Learn: Self-supervised learning from permutations via differentiable ranking
In particular, we also improve music understanding by reordering spectrogram patches in the frequency space, as well as video classification by reordering frames along the time axis.
A Universal Learnable Audio Frontend
Mel-filterbanks are fixed, engineered audio features which emulate human perception and have lived through the history of audio understanding up to today.
Contrastive Learning of General-Purpose Audio Representations
We introduce COLA, a self-supervised pre-training approach for learning a general-purpose representation of audio.
Ranked #4 on
Spoken Command Recognition
on Speech Command v2
Learning from Heterogeneous EEG Signals with Differentiable Channel Reordering
We propose CHARM, a method for training a single neural network across inconsistent input channels.
Wavesplit: End-to-End Speech Separation by Speaker Clustering
Wavesplit infers a set of source representations via clustering, which addresses the fundamental permutation problem of separation.
Ranked #6 on
Speech Separation
on WHAMR!
Deep multi-class learning from label proportions
We propose a learning algorithm capable of learning from label proportions instead of direct data labels.
Fully Convolutional Speech Recognition
In this paper we present an alternative approach based solely on convolutional neural networks, leveraging recent advances in acoustic models from the raw waveform and language modeling.
Ranked #3 on
Speech Recognition
on WSJ eval93
To Reverse the Gradient or Not: An Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition
In multi-task learning, the goal is speaker prediction; we expect a performance improvement with this joint training if the two tasks of speech recognition and speaker recognition share a common set of underlying features.
Learning to detect dysarthria from raw speech
We extend this approach to paralinguistic classification and propose a neural network that can learn a filterbank, a normalization factor and a compression power from the raw speech, jointly with the rest of the architecture.
SING: Symbol-to-Instrument Neural Generator
On the generalization task of synthesizing notes for pairs of pitch and instrument not seen during training, SING produces audio with significantly improved perceptual quality compared to a state-of-the-art autoencoder based on WaveNet as measured by a Mean Opinion Score (MOS), and is about 32 times faster for training and 2, 500 times faster for inference.
End-to-End Speech Recognition From the Raw Waveform
In this paper, we study end-to-end systems trained directly from the raw waveform, building on two alternatives for trainable replacements of mel-filterbanks that use a convolutional architecture.
Sampling strategies in Siamese Networks for unsupervised speech representation learning
We apply these results to pairs of words discovered using an unsupervised algorithm and show an improvement on state-of-the-art in unsupervised representation learning using siamese networks.
Fader Networks:Manipulating Images by Sliding Attributes
This paper introduces a new encoder-decoder architecture that is trained to reconstruct images by disentangling the salient information of the image and the values of attributes directly in the latent space.
Learning Filterbanks from Raw Speech for Phone Recognition
We train a bank of complex filters that operates on the raw waveform and is fed into a convolutional neural network for end-to-end phone recognition.
Fader Networks: Manipulating Images by Sliding Attributes
This paper introduces a new encoder-decoder architecture that is trained to reconstruct images by disentangling the salient information of the image and the values of attributes directly in the latent space.
Learning weakly supervised multimodal phoneme embeddings
Recent works have explored deep architectures for learning multimodal speech representation (e. g. audio and images, articulation and audio) in a supervised way.
