Search Results for author:
Found 105 papers, 53 papers with code
Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance
We propose a novel step-by-step video-to-audio generation method that sequentially produces individual audio tracks, each corresponding to a specific sound event in the video.
Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometry
We eliminate the need for extensive 4D training data through a factorized fine-tuning framework that separately trains spatial and temporal components using multi-view image and video data.
A Comprehensive Real-World Assessment of Audio Watermarking Algorithms: Will They Survive Neural Codecs?
We introduce the Robust Audio Watermarking Benchmark (RAW-Bench), a benchmark for evaluating deep learning-based audio watermarking methods with standardized and systematic comparisons.
SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet
ControlNet, a method for adding fine-grained controls to pretrained generative models, has been applied to foley synthesis, but its use has been limited to handcrafted human-readable temporal conditions.
Improving Inference-Time Optimisation for Vocal Effects Style Transfer with a Gaussian Prior
Style Transfer with Inference-Time Optimisation (ST-ITO) is a recent approach for transferring the applied effects of a reference audio to a raw audio track.
Dyadic Mamba: Long-term Dyadic Human Motion Synthesis
Generating realistic dyadic human motion from text descriptions presents significant challenges, particularly for extended interactions that exceed typical training sequence lengths.
Forging and Removing Latent-Noise Diffusion Watermarks Using a Single Image
Based on this intuition, we propose an adversarial attack to forge the watermark by introducing perturbations to the images such that we can enter the region of watermarked images.
D^2USt3R: Enhancing 3D Reconstruction with 4D Pointmaps for Dynamic Scenes
We address the task of 3D reconstruction in dynamic scenes, where object motions degrade the quality of previous 3D pointmap regression methods, such as DUSt3R, originally designed for static 3D scene reconstruction.
CARE: Aligning Language Models for Regional Cultural Awareness
Existing language models (LMs) often exhibit a Western-centric bias and struggle to represent diverse cultural knowledge.
VinaBench: Benchmark for Faithful and Consistent Visual Narratives
Visual narrative generation transforms textual narratives into sequences of images illustrating the content of the text.
Aligning Text-to-Music Evaluation with Human Preferences
In this work, we rigorously study the design space of reference-based divergence metrics for evaluating TTM models through (1) designing four synthetic meta-evaluations to measure sensitivity to particular musical desiderata, and (2) collecting and evaluating on MusicPrefs, the first open-source dataset of human preferences for TTM systems.
Cross-Modal Learning for Music-to-Music-Video Description Generation
In this study, we focus on the MV description generation task and propose a comprehensive pipeline encompassing training data construction and multimodal model fine-tuning.
Training Consistency Models with Variational Noise Coupling
Consistency Training (CT) has recently emerged as a promising alternative to diffusion models, achieving competitive performance in image generation tasks.
Supervised contrastive learning from weakly-labeled audio segments for musical version matching
Detecting musical versions (different renditions of the same piece) is a challenging task with important applications.
DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning
Recent advancements in music large language models (LLMs) have significantly improved music understanding tasks, which involve the model's ability to analyze and interpret various musical elements.
HumanGif: Single-View Human Diffusion with Generative Prior
While previous single-view-based 3D human reconstruction methods made significant progress in novel view synthesis, it remains a challenge to synthesize both view-consistent and pose-consistent results for animatable human avatars from a single image input.
30+ Years of Source Separation Research: Achievements and Future Challenges
Source separation (SS) of acoustic signals is a research field that emerged in the mid-1990s and has flourished ever since.
Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models
To reduce this gap and further improve the power of LoRA, we propose a new PEFT method that combines two classes of adaptations, namely, transform and residual adaptations.
CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation
Binaural audio generation (BAG) aims to convert monaural audio to stereo audio using visual prompts, requiring a deep understanding of spatial and semantic information.
MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis
We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework MMAudio.
Ranked #1 on
Video-to-Sound Generation
on VGG-Sound
TraSCE: Trajectory Steering for Concept Erasure
To address this issue, we first propose a modification of conventional negative prompting.
Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion
Blind inverse problems, where both the target data and forward operator are unknown, are crucial to many computer vision applications.
Classifier-Free Guidance inside the Attraction Basin May Cause Memorization
Diffusion models are prone to exactly reproduce images from the training data.
Music Foundation Model as Generic Booster for Music Downstream Tasks
This highlights the capability of using features extracted from music foundation models as a booster for downstream tasks.
OpenMU: Your Swiss Army Knife for Music Understanding
We present OpenMU-Bench, a large-scale benchmark suite for addressing the data scarcity issue in training multimodal language models to understand music.
Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusion
By embedding discrete representations into a continuous latent space, we can leverage continuous-space latent diffusion models to handle generative modeling of discrete data.
Distillation of Discrete Diffusion through Dimensional Correlations
Diffusion models have demonstrated exceptional performances in various fields of generative modeling, but suffer from slow sampling speed due to their iterative nature.
$\textit{Jump Your Steps}$: Optimizing Sampling Schedule of Discrete Diffusion Models
Diffusion models have seen notable success in continuous domains, leading to the development of discrete diffusion models (DDMs) for discrete variables.
G2D2: Gradient-guided Discrete Diffusion for image inverse problem solving
Recent literature has effectively utilized diffusion models trained on continuous variables as priors for solving inverse problems.
Variable Bitrate Residual Vector Quantization for Audio Coding
Recent state-of-the-art neural audio compression models have progressively adopted residual vector quantization (RVQ).
GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models
In each respective optimization step, the ranked prompts are fed as in-context examples (with their accuracies) to equip the LLM with the knowledge of the type of text prompts preferred by the downstream VLM.
HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning
Controllable generation through Stable Diffusion (SD) fine-tuning aims to improve fidelity, safety, and alignment with human guidance.
Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space
To address this, we introduce Bellman Diffusion, a novel DGM framework that maintains linearity in MDPs through gradient and scalar field modeling.
Mining Your Own Secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models
Personalized text-to-image diffusion models have grown popular for their ability to efficiently acquire a new concept from user-defined text descriptions and a few images.
A Survey on Diffusion Models for Inverse Problems
Diffusion models have become increasingly popular for generative modeling due to their ability to generate high-quality samples.
A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation
In CMC-PE, cross-modal information is embedded as if it represents temporal position information, and the embeddings are fed into the model like positional encoding.
Latent Diffusion Bridges for Unsupervised Musical Audio Timbre Transfer
Each diffusion model is trained on a specific instrument with a Gaussian prior.
DisMix: Disentangling Mixtures of Musical Instruments for Source-level Pitch and Timbre Manipulation
Existing work on pitch and timbre disentanglement has been mostly focused on single-instrument music audio, excluding the cases where multiple instruments are presented.
Towards Assessing Data Replication in Music Generation with Music Similarity Metrics on Raw Audio
Recent advancements in music generation are raising multiple concerns about the implications of AI in creative music processes, current business models and impacts related to intellectual property management.
ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark
We propose a new benchmark, ComperDial, which facilitates the training and evaluation of evaluation metrics for open-domain dialogue systems.
MoLA: Motion Generation and Editing with Latent Diffusion Enhanced by Adversarial Training
For high-quality and fast generation, we employ a variational autoencoder and latent diffusion model, and improve the performance with adversarial training.
SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation
To address these issues, we introduce Sound Consistency Trajectory Models (SoundCTM), which allow flexible transitions between high-quality $1$-step sound generation and superior sound quality through multi-step deterministic sampling.
MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation
Specifically, given two pre-trained base diffusion models, we train a lightweight joint guidance module to adjust scores separately estimated by the base models to match the score of joint distribution over audio and video.
Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning
Recent advances in text-to-music editing, which employ text queries to modify music (e. g.\ by changing its style or adjusting instrumental components), present unique challenges and opportunities for AI-assisted music creation.
GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping
In these methods, an input view is geometrically warped to novel views with estimated depth maps, then the warped image is inpainted by T2I models.
Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation
The recent audio-visual generation methods usually resort to huge large language model or composable diffusion models.
PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher
The diffusion model performs remarkable in generating high-dimensional content but is computationally intensive, especially during training.
Ranked #1 on
Image Generation
on ImageNet 32x32
Weighted Point Cloud Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric
In addition, we show that our proposed similarity based on weighted point clouds consistently achieves the optimal similarity.
Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation
Prompt engineering is effective for controlling the output of text-to-image (T2I) generative models, but it is also laborious due to the need for manually crafted prompts.
MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakage
This paper presents enhancements to the MT3 model, a state-of-the-art (SOTA) token-based multi-instrument automatic music transcription (AMT) model.
DiffuCOMET: Contextual Commonsense Knowledge Diffusion
Inferring contextually-relevant and diverse commonsense to understand narratives remains challenging for knowledge models.
MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models
This paper introduces a novel approach to the editing of music generated by such models, enabling the modification of specific attributes, such as genre, mood and instrument, while maintaining other aspects unchanged.
HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes
Vector quantization (VQ) is a technique to deterministically learn features with discrete codebook representations.
Manifold Preserving Guided Diffusion
Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training.
On the Language Encoder of Contrastive Cross-modal Models
Contrastive cross-modal models such as CLIP and CLAP aid various vision-language (VL) and audio-language (AL) tasks.
Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association
In this paper, we focus on the wide existence of reporting bias in visual-language datasets, embodied as the object-attribute association, which can subsequentially degrade models trained on them.
Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion
Consistency Models (CM) (Song et al., 2023) accelerate score-based diffusion model sampling at the cost of sample quality but lack a natural way to trade-off quality for speed.
Ranked #1 on
Image Generation
on ImageNet 64x64
(NFE metric)
Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music Transcription
Several works have explored multi-instrument transcription as a means to bolster the performance of models on low-resource tasks, but these methods face the same data availability issues.
VRDMG: Vocal Restoration via Diffusion Posterior Sampling with Multiple Guidance
Restoring degraded music signals is essential to enhance audio quality for downstream music manipulation.
BigVSAN: Enhancing GAN-based Neural Vocoders with Slicing Adversarial Network
In the literature, it has been demonstrated that slicing adversarial network (SAN), an improved GAN training framework that can find the optimal projection, is effective in the image generation task.
Ranked #6 on
Speech Synthesis
on LibriTTS
Enhancing Semantic Communication with Deep Generative Models -- An ICASSP Special Session Overview
Semantic communication is poised to play a pivotal role in shaping the landscape of future AI-driven communication systems.
The Sound Demixing Challenge 2023 $\unicode{x2013}$ Cinematic Demixing Track
A significant source of this improvement was making the simulated data better match real cinematic audio, which we further investigate in detail.
The Sound Demixing Challenge 2023 $\unicode{x2013}$ Music Demixing Track
We propose a formalization of the errors that can occur in the design of a training dataset for MSS systems and introduce two new datasets that simulate such errors: SDXDB23_LabelNoise and SDXDB23_Bleeding.
Automatic Piano Transcription with Hierarchical Frequency-Time Transformer
This is especially helpful when determining the precise onset and offset for each note in the polyphonic piano content.
Ranked #2 on
Music Transcription
on MAESTRO
STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events
While direction of arrival (DOA) of sound events is generally estimated from multichannel audio data recorded in a microphone array, sound events usually derive from visually perceptible source objects, e. g., sounds of footsteps come from the feet of a walker.
On the Equivalence of Consistency-Type Models: Consistency Models, Consistent Diffusion Models, and Fokker-Planck Regularization
The emergence of various notions of ``consistency'' in diffusion models has garnered considerable attention and helped achieve improved sample quality, likelihood estimation, and accelerated sampling.
Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders
At the decoded feature level, we fuse the two decoded features by generative and predictive decoders.
The Whole Is Greater than the Sum of Its Parts: Improving Music Source Separation by Bridging Network
We modify the target network, i. e., the network architecture of the original DNN-based MSS, by adding bridging paths for each output instrument to share their information.
Diffusion-based Signal Refiner for Speech Separation
We experimentally show that our refiner can provide a clearer harmonic structure of speech and improves the reference-free metric of perceptual quality for arbitrary preceding model architectures.
PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives
They must also learn to maintain consistent speaker personas for themselves throughout the narrative, so that their counterparts feel involved in a realistic conversation or story.
Cross-modal Face- and Voice-style Transfer
Image-to-image translation and voice conversion enable the generation of a new facial image and voice while maintaining some of the semantics such as a pose in an image and linguistic content in audio, respectively.
SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer
Generative adversarial networks (GANs) learn a target probability distribution by optimizing a generator and a discriminator with minimax objectives.
Ranked #1 on
Image Generation
on FFHQ 1024 x 1024
GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration
Pre-trained diffusion models have been successfully used as priors in a variety of linear inverse problems, where the goal is to reconstruct a signal from noisy linear measurements.
CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos
Further, videos in the wild often contain off-screen sounds and background noise that may hinder the model from learning the desired audio-textual correspondence.
Unsupervised vocal dereverberation with diffusion-based generative models
Removing reverb from reverberant music is a necessary technique to clean up audio for downstream music manipulations.
Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects
We propose an end-to-end music mixing style transfer system that converts the mixing style of an input multitrack to that of a reference song.
Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement
Although deep neural network (DNN)-based speech enhancement (SE) methods outperform the previous non-DNN-based ones, they often degrade the perceptual quality of generated outputs.
ComFact: A Benchmark for Linking Contextual Commonsense Knowledge
Understanding rich narratives, such as dialogues and stories, often requires natural language processing systems to access relevant knowledge from commonsense knowledge graphs.
Robust One-Shot Singing Voice Conversion
We then propose a two-stage training method called Robustify that train the one-shot SVC model in the first stage on clean data to ensure high-quality conversion, and introduces enhancement modules to the encoders of the model in the second stage to enhance the feature extraction from distorted singing voices.
Hierarchical Diffusion Models for Singing Voice Neural Vocoder
Recent progress in deep generative models has improved the quality of neural vocoders in speech domain.
DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability
In this paper we propose a novel generative approach, DiffRoll, to tackle automatic music transcription (AMT).
FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation
Score-based generative models (SGMs) learn a family of noise-conditional score functions corresponding to the data density perturbed with increasingly large amounts of noise.
Automatic music mixing with deep learning and out-of-domain data
Music mixing traditionally involves recording instruments in the form of clean, individual tracks and blending them into a final mixture using audio effects and expert knowledge (e. g., a mixing engineer).
STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events
Additionally, the report presents the baseline system that accompanies the dataset in the challenge with emphasis on the differences with the baseline of the previous iterations; namely, introduction of the multi-ACCDOA representation to handle multiple simultaneous occurences of events of the same class, and support for additional improved input features for the microphone array format.
Ranked #1 on
Sound Event Localization and Detection
on STARSS22
SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization
In this paper, we propose a new training scheme that extends the standard VAE via novel stochastic dequantization and quantization, called stochastically quantized variational autoencoder (SQ-VAE).
Distortion Audio Effects: Learning How to Recover the Clean Signal
Given the recent advances in music source separation and automatic mixing, removing audio effects in music tracks is a meaningful step toward developing an automated remixing system.
Multi-ACCDOA: Localizing and Detecting Overlapping Sounds from the Same Class with Auxiliary Duplicating Permutation Invariant Training
The multi- ACCDOA format (a class- and track-wise output format) enables the model to solve the cases with overlaps from the same class.
Automatic DJ Transitions with Differentiable Audio Effects and Generative Adversarial Networks
A central task of a Disc Jockey (DJ) is to create a mixset of mu-sic with seamless transitions between adjacent tracks.
Spatial mixup: Directional loudness modification as data augmentation for sound event localization and detection
Data augmentation methods have shown great importance in diverse supervised learning problems where labeled data is scarce or costly to obtain.
Music Demixing Challenge 2021
The main differences compared with the past challenges are 1) the competition is designed to more easily allow machine learning practitioners from other disciplines to participate, 2) evaluation is done on a hidden test set created by music professionals dedicated exclusively to the challenge to assure the transparency of the challenge, i. e., the test set is not accessible from anyone except the challenge organizers, and 3) the dataset provides a wider range of music genres and involved a greater number of mixing engineers.
Ensemble of ACCDOA- and EINV2-based Systems with D3Nets and Impulse Response Simulation for Sound Event Localization and Detection
This report describes our systems submitted to the DCASE2021 challenge task 3: sound event localization and detection (SELD) with directional interference.
Densely Connected Multi-Dilated Convolutional Networks for Dense Prediction Tasks
In this paper, we claim the importance of a dense simultaneous modeling of multiresolution representation and propose a novel CNN architecture called densely connected multidilated DenseNet (D3Net).
Training Speech Enhancement Systems with Noisy Speech Datasets
Furthermore, we propose a noise augmentation scheme for mixture-invariant training (MixIT), which allows using it also in such scenarios.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Preventing Oversmoothing in VAE via Generalized Variance Parameterization
Variational autoencoders (VAEs) often suffer from posterior collapse, which is a phenomenon in which the learned latent space becomes uninformative.
Hierarchical disentangled representation learning for singing voice conversion
Conventional singing voice conversion (SVC) methods often suffer from operating in high-resolution audio owing to a high dimensionality of data.
AR-ELBO: Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE
Variational autoencoders (VAEs) often suffer from posterior collapse, which is a phenomenon that the learned latent space becomes uninformative.
Densely connected multidilated convolutional networks for dense prediction tasks
In this paper, we claim the importance of a dense simultaneous modeling of multiresolution representation and propose a novel CNN architecture called densely connected multidilated DenseNet (D3Net).
Ranked #48 on
Semantic Segmentation
on Cityscapes test
ACCDOA: Activity-Coupled Cartesian Direction of Arrival Representation for Sound Event Localization and Detection
Conventional NN-based methods use two branches for a sound event detection (SED) target and a direction-of-arrival (DOA) target.
All for One and One for All: Improving Music Separation by Bridging Networks
This paper proposes several improvements for music separation with deep neural networks (DNNs), namely a multi-domain loss (MDL) and two combination schemes.
Ranked #21 on
Music Source Separation
on MUSDB18
Adversarial attacks on audio source separation
Despite the excellent performance of neural-network-based audio source separation methods and their wide range of applications, their robustness against intentional attacks has been largely neglected.
D3Net: Densely connected multidilated DenseNet for music source separation
In this paper, we claim the importance of a rapid growth of a receptive field and a simultaneous modeling of multi-resolution data in a single convolution layer, and propose a novel CNN architecture called densely connected dilated DenseNet (D3Net).
Ranked #12 on
Music Source Separation
on MUSDB18
(using extra training data)
Improving Voice Separation by Incorporating End-to-end Speech Recognition
Despite recent advances in voice separation methods, many challenges remain in realistic scenarios such as noisy recording and the limits of available data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Improving DNN-based Music Source Separation using Phase Features
Music source separation with deep neural networks typically relies only on amplitude features.
MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation
Deep neural networks have become an indispensable technique for audio source separation (ASS).
Ranked #17 on
Music Source Separation
on MUSDB18
(using extra training data)
Music Source Separation
Sound
Audio and Speech Processing
Multi-scale Multi-band DenseNets for Audio Source Separation
This paper deals with the problem of audio source separation.
