Search Results for author:
Found 16 papers, 7 papers with code
Video-Guided Foley Sound Generation with Multimodal Controls
MultiFoley also allows users to choose reference audio from sound effects (SFX) libraries or partial videos for conditioning.
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
The ability to comprehend audio--which includes speech, non-speech sounds, and music--is crucial for AI agents to interact effectively with the world.
Augment, Drop & Swap: Improving Diversity in LLM Captions for Efficient Music-Text Representation Learning
Audio-text contrastive models have become a powerful approach in music representation learning.
ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds
To achieve this, we first propose ReCLAP, a CLAP model trained with rewritten audio captions for improved understanding of sounds in the wild.
GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities
Next, we propose CompA-R (Instruction-Tuning for Complex Audio Reasoning), a synthetically generated instruction-tuning (IT) dataset with instructions that require the model to perform complex reasoning on the input audio.
Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs
To address this gap, we introduce Visual Description Grounded Decoding (VDGD), a simple, robust, and training-free method designed to enhance visual perception and improve reasoning capabilities in LVLMs.
CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models
In this paper, we propose CompA, a collection of two expert-annotated benchmarks with a majority of real-world audio samples, to evaluate compositional reasoning in ALMs.
Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queries
We show that our system, trained using our automatic data curation pipeline, significantly outperforms baselines trained on in-the-wild data on the task of HQ SFX retrieval for video.
Efficient Spoken Language Recognition via Multilabel Classification
Spoken language recognition (SLR) is the task of automatically identifying the language present in a speech signal.
Language-Guided Audio-Visual Source Separation via Trimodal Consistency
We propose a self-supervised approach for learning to perform audio source separation in videos based on natural language queries, using only unlabeled video and audio pairs as training data.
Music Enhancement via Image Translation and Vocoding
Consumer-grade music recordings such as those captured by mobile devices typically contain distortions in the form of background noise, reverb, and microphone-induced EQ.
Multimodal Metric Learning for Tag-based Music Retrieval
In this paper, we investigate three ideas to successfully introduce multimodal metric learning for tag-based music retrieval: elaborate triplet sampling, acoustic and cultural music information, and domain-specific word embeddings.
Mood Classification Using Listening Data
The mood of a song is a highly relevant feature for exploration and recommendation in large collections of music.
Predicting Audio Advertisement Quality
In these platforms, which tend to host tens of thousands of unique audio advertisements (ads), providing high quality ads ensures a better user experience and results in longer user engagement.
End-to-end learning for music audio tagging at scale
The lack of data tends to limit the outcomes of deep learning research, particularly when dealing with end-to-end learning stacks processing raw data such as waveforms.
Sound Audio and Speech Processing
A Deep Multimodal Approach for Cold-start Music Recommendation
Second, track embeddings are learned from the audio signal and available feedback data.
