Search Results for author:
Found 30 papers, 14 papers with code
Multi-Domain Audio Question Answering Toward Acoustic Content Reasoning in The DCASE 2025 Challenge
We present Task 5 of the DCASE 2025 Challenge: an Audio Question Answering (AQA) benchmark spanning multiple domains of sound understanding.
Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed LongAudioBench, an expert annotated benchmark for evaluating ALMs on long audio understanding capabilities.
UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation
We show that with the appropriate design choices for pre-training, one can jointly learn a representation encoder and generative audio decoder that can be applied to both types of tasks.
Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
While autoregressive speech token generation models produce speech with remarkable variety and naturalness, their inherent lack of controllability often results in issues such as hallucinations and undesired vocalizations that do not conform to conditioning inputs.
A2SB: Audio-to-Audio Schrodinger Bridges
Audio in the real world may be perturbed due to numerous factors, causing the audio quality to be degraded.
TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization
We introduce TangoFlux, an efficient Text-to-Audio (TTA) generative model with 515M parameters, capable of generating up to 30 seconds of 44. 1kHz audio in just 3. 7 seconds on a single A40 GPU.
Ranked #2 on
Audio Generation
on AudioCaps
ETTA: Elucidating the Design Space of Text-to-Audio Models
Recent years have seen significant progress in Text-To-Audio (TTA) synthesis, enabling users to enrich their creative workflows with synthetic audio generated from natural language prompts.
Ranked #1 on
Audio Generation
on AudioCaps
OMCAT: Omni Context Aware Transformer
OCTAV (Omni Context and Temporal Audio Video) is a novel dataset designed to capture event transitions across audio and video.
Audio-visual Question Answering
Audio-Visual Question Answering (AVQA)
+5
Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data
We present Synthio, a novel approach for augmenting small-scale audio classification datasets with synthetic data.
Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment
Large Language Model (LLM) based text-to-speech (TTS) systems have demonstrated remarkable capabilities in handling large speech datasets and generating natural speech for new speakers.
Improving Text-To-Audio Models with Synthetic Captions
It is an open challenge to obtain high quality training data, especially captions, for text-to-audio models.
Ranked #5 on
Audio Generation
on AudioCaps
(CLAP_LAION metric)
Audio Dialogues: Dialogues dataset for audio and music understanding
Existing datasets for audio understanding primarily focus on single-turn interactions (i. e. audio captioning, audio question answering) for describing audio in natural language, thus limiting understanding audio via interactive dialogue.
Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs.
Ranked #1 on
Retrieval-augmented Few-shot In-context Audio Captioning
on AudioCaps
(using extra training data)
Scaling NVIDIA's Multi-speaker Multi-lingual TTS Systems with Zero-Shot TTS to Indic Languages
In Track 3, we utilize P-Flow to perform zero-shot TTS by training on the challenge dataset as well as external datasets.
SelfVC: Voice Conversion With Iterative Refinement using Self Transformations
In this work, instead of explicitly disentangling attributes with loss terms, we present a framework to train a controllable voice conversion model on entangled speech representations derived from self-supervised learning (SSL) and speaker verification models.
P-Flow: A Fast and Data-Efficient Zero-Shot TTS through Speech Prompting
P-Flow comprises a speech-prompted text encoder for speaker adaptation and a flow matching generative decoder for high-quality and fast speech synthesis.
VANI: Very-lightweight Accent-controllable TTS for Native and Non-native speakers with Identity Preservation
We introduce VANI, a very lightweight multi-lingual accent controllable speech synthesis system.
Multilingual Multiaccented Multispeaker TTS with RADTTS
We work to create a multilingual speech synthesis system which can generate speech with the proper accent while retaining the characteristics of an individual voice.
SPACE: Speech-driven Portrait Animation with Controllable Expression
It uses a multi-stage approach, combining the controllability of facial landmarks with the high-quality synthesis power of a pretrained face generator.
Generative Modeling for Low Dimensional Speech Attributes with Neural Spline Flows
Despite recent advances in generative modeling for text-to-speech synthesis, these models do not yet have the same fine-grained adjustability of pitch-conditioned deterministic models such as FastPitch and FastSpeech2.
One TTS Alignment To Rule Them All
However, these alignments tend to be brittle and often fail to generalize to long utterances and out-of-domain text, leading to missing or repeating words.
Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
In this paper we propose Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis with control over speech variation and style transfer.
Ranked #1 on
Text-To-Speech Synthesis
on LJSpeech
(Pleasantness MOS metric, using extra
training data)
Neural ODEs for Image Segmentation with Level Sets
We propose a novel approach for image segmentation that combines Neural Ordinary Differential Equations (NODEs) and the Level Set method.
Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens
Mellotron is a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data.
WaveGlow: A Flow-based Generative Network for Speech Synthesis
In this paper we propose WaveGlow: a flow-based network capable of generating high quality speech from mel-spectrograms.
Ranked #12 on
Speech Synthesis
on LibriTTS
TequilaGAN: How to easily identify GAN samples
In this paper we show strategies to easily identify fake samples generated with the Generative Adversarial Network framework.
Attacking Speaker Recognition With Deep Generative Models
In this paper we investigate the ability of generative adversarial networks (GANs) to synthesize spoofing attacks on modern speaker recognition systems.
Character-Based Handwritten Text Transcription with Attention Networks
When the sequence alignment is one-to-one, softmax attention is able to learn a more precise alignment at each step of the decoding, whereas the alignment generated by sigmoid attention is much less precise.
Missing Data Imputation for Supervised Learning
Missing data imputation can help improve the performance of prediction models in situations where missing data hide useful information.
Ranked #1 on
Imputation
on Adult
