Search Results for author:
Found 27 papers, 8 papers with code
Long-Form Speech Generation with Spoken Language Models
We consider the generative modeling of speech over multiple minutes, a requirement for long-form multimedia generation and audio-native voice assistants.
MELODI: Exploring Memory Compression for Long Contexts
We present MELODI, a novel memory architecture designed to efficiently process long documents using short context windows.
A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation
Here, we propose a novel training approach to effectively learn arbitrary conditional distributions in the audiovisual space. Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process.
Dataset balancing can hurt model performance
We find, however, that while balancing improves performance on the public AudioSet evaluation data it simultaneously hurts performance on an unpublished evaluation set collected under the same conditions.
V2Meow: Meowing to the Visual Beat via Video-to-Music Generation
Video-to-music generation demands both a temporally localized high-quality listening experience and globally aligned video-acoustic signatures.
MusicLM: Generating Music From Text
We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff".
Ranked #15 on
Text-to-Music Generation
on MusicCaps
MAQA: A Multimodal QA Benchmark for Negation
Multimodal learning can benefit from the representation power of pretrained Large Language Models (LLMs).
MuLan: A Joint Embedding of Music Audio and Natural Language
Music tagging and content-based retrieval systems have traditionally been constructed using pre-defined ontologies covering a rigid set of music attributes or text queries.
Text-Driven Separation of Arbitrary Sounds
The second model, SoundFilter, takes a mixed source audio clip as an input and separates it based on a conditioning vector from the shared text-audio representation defined by SoundWords, making the model agnostic to the conditioning modality.
Universal Paralinguistic Speech Representations Using Self-Supervised Conformers
Many speech applications require understanding aspects beyond the words being spoken, such as recognizing emotion, detecting whether the speaker is wearing a mask, or distinguishing real from synthetic speech.
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio.
Ranked #1 on
Speech Recognition
on Common Voice
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Attention Bottlenecks for Multimodal Fusion
Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio.
Ranked #2 on
Action Classification
on Kinetics-Sounds
Sparse, Efficient, and Semantic Mixture Invariant Training: Taming In-the-Wild Unsupervised Sound Separation
The best performance is achieved using larger numbers of output sources, enabled by our efficient MixIT loss, combined with sparsity losses to prevent over-separation.
Self-Supervised Learning from Automatically Separated Sound Scenes
Real-world sound scenes consist of time-varying collections of sound sources, each generating characteristic sound events that are mixed together in audio recordings.
Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds
For evaluation and semi-supervised experiments, we collected human labels for presence of on-screen and off-screen sounds on a small subset of clips.
Addressing Missing Labels in Large-Scale Sound Event Recognition Using a Teacher-Student Framework With Loss Masking
The study of label noise in sound event recognition has recently gained attention with the advent of larger and noisier datasets.
Towards Learning a Universal Non-Semantic Representation of Speech
The ultimate goal of transfer learning is to reduce labeled data requirements by exploiting a pre-existing embedding model trained for different datasets or tasks.
Improving Universal Sound Separation Using Sound Classification
Deep learning approaches have recently achieved impressive performance on both audio source separation and sound classification.
Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision
Humans do not acquire perceptual abilities in the way we train machines.
Unsupervised Learning of Semantic Audio Representations
Even in the absence of any explicit semantic annotation, vast collections of audio recordings provide valuable information for learning the categorical structure of sounds.
Ranked #50 on
Audio Classification
on AudioSet
Evaluating Low-Level Speech Features Against Human Perceptual Data
We introduce a method for measuring the correspondence between low-level speech features and human perception, using a cognitive model of speech perception implemented directly on speech recordings.
Automatic Speech Recognition (ASR)
Representation Learning
+1
CNN Architectures for Large-Scale Audio Classification
Convolutional Neural Networks (CNNs) have proven very effective in image classification and show promise for audio.
A segmental framework for fully-unsupervised large-vocabulary speech recognition
We also show that the discovered clusters can be made less speaker- and gender-specific by using an unsupervised autoencoder-like feature extractor to learn better frame-level features (prior to embedding).
Unsupervised word segmentation and lexicon discovery using acoustic word embeddings
In settings where only unlabelled speech data is available, speech technology needs to be developed without transcriptions, pronunciation dictionaries, or language modelling text.
Scalable Out-of-Sample Extension of Graph Embeddings Using Deep Neural Networks
Several popular graph embedding techniques for representation learning and dimensionality reduction rely on performing computationally expensive eigendecompositions to derive a nonlinear transformation of the input data space.
Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems
The unsupervised discovery of linguistic terms from either continuous phoneme transcriptions or from raw speech has seen an increasing interest in the past years both from a theoretical and a practical standpoint.
