Search Results for author:
Found 33 papers, 13 papers with code
MAD Speech: Measures of Acoustic Diversity of Speech
Using these datasets, we demonstrate that our proposed metrics achieve a stronger agreement with the ground-truth diversity than baselines.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
In this report, we present the latest model of the Gemini family, Gemini 1. 5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio.
Ranked #20 on
Code Generation
on HumanEval
TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition
The model operates on transcripts and audio token sequences and achieves multiple tasks through masking of inputs.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
AudioPaLM: A Large Language Model That Can Speak and Listen
AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2.
SoundStorm: Efficient Parallel Audio Generation
We present SoundStorm, a model for efficient, non-autoregressive audio generation.
MusicLM: Generating Music From Text
We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff".
Ranked #8 on
Text-to-Music Generation
on MusicCaps
AudioLM: a Language Modeling Approach to Audio Generation
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency.
Text-Driven Separation of Arbitrary Sounds
The second model, SoundFilter, takes a mixed source audio clip as an input and separates it based on a conditioning vector from the shared text-audio representation defined by SoundWords, making the model agnostic to the conditioning modality.
Disentangling speech from surroundings with neural embeddings
We present a method to separate speech signals from noisy environments in the embedding space of a neural audio codec.
CycleGAN-Based Unpaired Speech Dereverberation
Typically, neural network-based speech dereverberation models are trained on paired data, composed of a dry utterance and its corresponding reverberant utterance.
Real time spectrogram inversion on mobile phone
We present two methods of real time magnitude spectrogram inversion: streaming Griffin Lim(GL) and streaming MelGAN.
SpeechPainter: Text-conditioned Speech Inpainting
We propose SpeechPainter, a model for filling in gaps of up to one second in speech samples by leveraging an auxiliary textual input.
Data Summarization via Bilevel Optimization
We show the effectiveness of our framework for a wide range of models in various settings, including training non-convex models online and batch active learning.
SoundStream: An End-to-End Neural Audio Codec
We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs.
Self-Supervised Learning from Automatically Separated Sound Scenes
Real-world sound scenes consist of time-varying collections of sound sources, each generating characteristic sound events that are mixed together in audio recordings.
LEAF: A Learnable Frontend for Audio Classification
In this work we show that we can train a single learnable frontend that outperforms mel-filterbanks on a wide range of audio signals, including speech, music, audio events and animal sounds, providing a general-purpose learned frontend for audio classification.
A Universal Learnable Audio Frontend
Mel-filterbanks are fixed, engineered audio features which emulate human perception and have lived through the history of audio understanding up to today.
One-shot conditional audio filtering of arbitrary sounds
When trained on Librispeech, our model achieves an SI-SDR improvement of 14. 0 dB when separating one voice from a mixture of two speakers.
Semi-supervised Batch Active Learning via Bilevel Optimization
Active learning is an effective technique for reducing the labeling cost by improving data efficiency.
MicAugment: One-shot Microphone Style Transfer
A crucial aspect for the successful deployment of audio-based models "in-the-wild" is the robustness to the transformations introduced by heterogeneous acquisition conditions.
SEANet: A Multi-modal Speech Enhancement Network
We explore the possibility of leveraging accelerometer data to perform speech enhancement in very noisy conditions.
Learning to Denoise Historical Music
We propose an audio-to-audio neural network model that learns to denoise old music recordings.
Towards Learning a Universal Non-Semantic Representation of Speech
The ultimate goal of transfer learning is to reduce labeled data requirements by exploiting a pre-existing embedding model trained for different datasets or tasks.
Learning audio representations via phase prediction
We learn audio representations by solving a novel self-supervised learning task, which consists of predicting the phase of the short-time Fourier transform from its magnitude.
SPICE: Self-supervised Pitch Estimation
We propose a model to estimate the fundamental frequency in monophonic audio, often referred to as pitch estimation.
From Here to There: Video Inbetweening Using Direct 3D Convolutions
We first obtain a latent video representation using a stochastic fusion mechanism that learns how to incorporate information from the start and end frames.
Self-supervised audio representation learning for mobile devices
We explore self-supervised models that can be potentially deployed on mobile devices to learn general purpose audio representations.
Deep convolutional neural networks for pedestrian detection
Pedestrian detection is a popular research topic due to its paramount importance for a number of applications, especially in the fields of automotive, surveillance and robotics.
Estimating snow cover from publicly available images
In this paper we study the problem of estimating snow cover in mountainous regions, that is, the spatial extent of the earth surface covered by snow.
SnowWatch: Snow Monitoring through Acquisition and Analysis of User-Generated Content
We present a system for complementing snow phenomena monitoring with virtual measurements extracted from public visual content.
Fast keypoint detection in video sequences
A number of computer vision tasks exploit a succinct representation of the visual content in the form of sets of local features.
Hybrid coding of visual content and local image features
Traditionally, a Compress-Then-Analyze approach has been pursued, in which sensing nodes acquire and encode the pixel-level representation of the visual content, that is subsequently transmitted to a sink node in order to be processed.
Coding local and global binary visual features extracted from video sequences
In this paper we investigate a coding scheme tailored to both local and global binary features, which aims at exploiting both spatial and temporal redundancy by means of intra- and inter-frame coding.
