Search Results for author:
Found 39 papers, 13 papers with code
Retrieval-Augmented Neural Field for HRTF Upsampling and Personalization
Head-related transfer functions (HRTFs) with dense spatial grids are desired for immersive binaural audio generation, but their recording is time-consuming.
30+ Years of Source Separation Research: Achievements and Future Challenges
Source separation (SS) of acoustic signals is a research field that emerged in the mid-1990s and has flourished ever since.
Meta-Learning for Physically-Constrained Neural System Identification
We present a gradient-based meta-learning framework for rapid adaptation of neural state-space models (NSSMs) for black-box system identification.
Task-Aware Unified Source Separation
These models are trained on large-scale data including speech, instruments, or sound events and can often successfully separate a wide range of sources.
Leveraging Audio-Only Data for Text-Queried Target Sound Extraction
A straightforward way to do so is to use a joint audio-text embedding model, such as the contrastive language-audio pre-training (CLAP) model, as a query encoder and train a TSE model using audio embeddings obtained from the ground-truth audio.
Enhanced Reverberation as Supervision for Unsupervised Speech Separation
Separation performance is also boosted by adding a novel loss term where separated signals mapped back to their own input mixture are used as pseudo-targets for the signals separated from other channels and mapped to the same channel.
TF-Locoformer: Transformer with Local Modeling by Convolution for Speech Separation and Enhancement
This work presents TF-Locoformer, a Transformer-based model with LOcal-modeling by COnvolution.
Ranked #1 on
Speech Separation
on WHAMR!
Sound Event Bounding Boxes
In this paper, we show that frame-level thresholding degrades the prediction of the event extent by coupling it with the system's sound presence confidence.
Why does music source separation benefit from cacophony?
In music source separation, a standard training data augmentation procedure is to create new training samples by randomly combining instrument stems from different songs.
NIIRF: Neural IIR Filter Field for HRTF Upsampling and Personalization
Existing NF-based methods focused on estimating the magnitude of the HRTF from a given sound source direction, and the magnitude is converted to a finite impulse response (FIR) filter.
NeuroHeed+: Improving Neuro-steered Speaker Extraction with Joint Auditory Attention Detection
Neuro-steered speaker extraction aims to extract the listener's brain-attended speech signal from a multi-talker speech signal, in which the attention is derived from the cortical activity.
Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction
Target speech extraction aims to extract, based on a given conditioning cue, a target speech signal that is corrupted by interfering sources, such as noise or competing speakers.
Generation or Replication: Auscultating Audio Latent Diffusion Models
The introduction of audio latent diffusion models possessing the ability to generate realistic sound clips on demand from a text description has the potential to revolutionize how we work with audio.
The Sound Demixing Challenge 2023 $\unicode{x2013}$ Cinematic Demixing Track
A significant source of this improvement was making the simulated data better match real cinematic audio, which we further investigate in detail.
Pac-HuBERT: Self-Supervised Music Source Separation via Primitive Auditory Clustering and Hidden-Unit BERT
In this paper, we propose a self-supervised learning framework for music source separation inspired by the HuBERT speech representation model.
TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings
Since diarization and source separation of meeting data are closely related tasks, we here propose an approach to perform the two objectives jointly.
Ranked #1 on
Speech Recognition
on LibriCSS
(using extra training data)
Tackling the Cocktail Fork Problem for Separation and Transcription of Real-World Soundtracks
In this paper, we focus on the cocktail fork problem, which takes a three-pronged approach to source separation by separating an audio mixture such as a movie soundtrack or podcast into the three broad categories of speech, music, and sound effects (SFX - understood to include ambient noise and natural sound events).
Hyperbolic Audio Source Separation
We introduce a framework for audio source separation using embeddings on a hyperbolic manifold that compactly represent the hierarchical relationship between sound sources and time-frequency features.
Latent Iterative Refinement for Modular Source Separation
During inference, we can dynamically adjust how many processing blocks and iterations of a specific block an input signal needs using a gating module.
Reverberation as Supervision for Speech Separation
This paper proposes reverberation as supervision (RAS), a novel unsupervised loss function for single-channel reverberant speech separation.
Meta-Learning of Neural State-Space Models Using Data From Similar Systems
Deep neural state-space models (SSMs) provide a powerful tool for modeling dynamical systems solely using operational data.
Optimal Condition Training for Target Source Separation
Recent research has shown remarkable performance in leveraging multiple extraneous conditional and non-mutually exclusive semantic concepts for sound source separation, allowing the flexibility to extract a given target source based on multiple different queries.
Cold Diffusion for Speech Enhancement
Diffusion models have recently shown promising results for difficult enhancement tasks such as the conditional and unconditional restoration of natural images and audio signals.
Late Audio-Visual Fusion for In-The-Wild Speaker Diarization
Speaker diarization is well studied for constrained audios but little explored for challenging in-the-wild videos, which have more speakers, shorter utterances, and inconsistent on-screen speakers.
Heterogeneous Target Speech Separation
We introduce a new paradigm for single-channel target source separation where the sources of interest can be distinguished using non-mutually exclusive concepts (e. g., loudness, gender, language, spatial location, etc).
Locate This, Not That: Class-Conditioned Sound Event DOA Estimation
Existing systems for sound event localization and detection (SELD) typically operate by estimating a source location for all classes at every time instant.
The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtracks
The cocktail party problem aims at isolating any source of interest within a complex acoustic scene, and has long inspired audio source separation research.
Attentive Neural Processes and Batch Bayesian Optimization for Scalable Calibration of Physics-Informed Digital Twins
Physics-informed dynamical system models form critical components of digital twins of the built environment.
Transcription Is All You Need: Learning to Separate Musical Mixtures with Score as Supervision
Most music source separation systems require large collections of isolated sources for training, which can be difficult to obtain.
AutoClip: Adaptive Gradient Clipping for Source Separation Networks
Clipping the gradient is a known approach to improving gradient descent, but requires hand selection of a clipping threshold hyperparameter.
Finding Strength in Weakness: Learning to Separate Sounds with Weak Supervision
In this scenario, weak labels are defined in contrast with strong time-frequency (TF) labels such as those obtained from isolated sources, and refer either to frame-level weak labels where one only has access to the time periods when different sources are active in an audio mixture, or to clip-level weak labels that only indicate the presence or absence of sounds in an entire audio clip.
Bootstrapping deep music separation from primitive auditory grouping principles
They are trained on synthetic mixtures of audio made from isolated sound source recordings so that ground truth for the separation is known.
WHAMR!: Noisy and Reverberant Single-Channel Speech Separation
While significant advances have been made with respect to the separation of overlapping speech signals, studies have been largely constrained to mixtures of clean, near anechoic speech, not representative of many real-world scenarios.
Sound Audio and Speech Processing
Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity
In this paper, we present the synthesized Lakh dataset (Slakh) as a new tool for music source separation research.
WHAM!: Extending Speech Separation to Noisy Environments
Recent progress in separating the speech signals from multiple overlapping speakers using a single audio channel has brought us closer to solving the cocktail party problem.
Ranked #18 on
Speech Separation
on WHAMR!
Class-conditional embeddings for music source separation
Isolating individual instruments in a musical mixture has a myriad of potential applications, and seems imminently achievable given the levels of performance reached by recent deep learning methods.
Bootstrapping single-channel source separation via unsupervised spatial clustering on stereo mixtures
These estimates, together with a weighting scheme in the time-frequency domain, based on confidence in the separation quality, are used to train a deep learning model that can be used for single-channel separation, where no source direction information is available.
Phasebook and Friends: Leveraging Discrete Representations for Source Separation
Here, we propose "magbook", "phasebook", and "combook", three new types of layers based on discrete representations that can be used to estimate complex time-frequency masks.
End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
We introduce a new dataset of dialogs about videos of human behaviors.
