Search Results for author:
Found 56 papers, 40 papers with code
Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
Recent advances in text-based large language models (LLMs), particularly in the GPT series and the o1 model, have demonstrated the effectiveness of scaling both training-time and inference-time compute.
Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoders
Automatic Music Transcription (AMT), aiming to get musical notes from raw audio, typically uses frame-level systems with piano-roll outputs or language model (LM)-based systems with note-level predictions.
DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
By tailoring the text embedding support and the caption datastore to the target domain, DRCap acquires a robust ability to adapt to new domains in a training-free manner.
Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement
Diffusion-based generative models have recently achieved remarkable results in speech and vocal enhancement due to their ability to model complex speech data distributions.
Language-Queried Target Sound Extraction Without Parallel Training Data
In a vanilla parallel-data-free training stage, target audio is encoded using the pre-trained CLAP audio encoder to form a condition embedding, while during testing, user language queries are encoded by CLAP text encoder as the condition embedding.
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
By enhancing the semantic ability of the codec, X-Codec significantly reduces WER in speech synthesis tasks and extends these benefits to non-speech applications, including music and sound generation.
Foundation Models for Music: A Survey
In recent years, foundation models (FMs) such as large language models (LLMs) and latent diffusion models (LDMs) have profoundly impacted diverse sectors, including music.
Universal Sound Separation with Self-Supervised Audio Masked Autoencoder
In this paper, we propose integrating a self-supervised pre-trained model, namely the audio masked autoencoder (A-MAE), into a universal sound separation system to enhance its separation performance.
Towards Out-of-Distribution Detection in Vocoder Recognition via Latent Feature Reconstruction
To enhance the distinctiveness of the reconstructed features by each decoder, we incorporate contrastive learning and an auxiliary classifier to further constrain the reconstructed feature.
WavCraft: Audio Editing and Generation with Large Language Models
We introduce WavCraft, a collective system that leverages large language models (LLMs) to connect diverse task-specific models for audio content creation and editing.
Selective-Memory Meta-Learning with Environment Representations for Sound Event Localization and Detection
In addition, we introduce environment representations to characterize different acoustic settings, enhancing the adaptability of our attenuation approach to various environments.
Joint Music and Language Attention Models for Zero-shot Music Tagging
However, previous music tagging research primarily focuses on close-set music tagging tasks which can not be generalized to new tags.
MERTech: Instrument Playing Technique Detection Using Self-Supervised Pretrained Model With Multi-Task Finetuning
Recognizing the significance of pitch in capturing the nuances of IPTs and the importance of onset in locating IPT events, we investigate multi-task finetuning with pitch and onset detection as auxiliary tasks.
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
Any audio can be translated into LOA based on AudioMAE, a self-supervised pre-trained representation learning model.
Ranked #4 on
Audio Generation
on AudioCaps
(FAD metric)
Separate Anything You Describe
In this work, we introduce AudioSep, a foundation model for open-domain audio source separation with natural language queries.
WavJourney: Compositional Audio Creation with Large Language Models
Subjective evaluations demonstrate the potential of WavJourney in crafting engaging storytelling audio content from text.
A unified front-end framework for English text-to-speech synthesis
The front-end is a critical component of English text-to-speech (TTS) systems, responsible for extracting linguistic features that are essential for a text-to-speech model to synthesize speech, such as prosodies and phonemes.
Multi-level Temporal-channel Speaker Retrieval for Zero-shot Voice Conversion
Specifically, to flexibly adapt to the dynamic-variant speaker characteristic in the temporal and channel axis of the speech, we propose a novel fine-grained speaker modeling method, called temporal-channel retrieval (TCR), to find out when and where speaker information appears in speech.
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions.
Ranked #1 on
Zero-Shot Environment Sound Classification
on ESC-50
(using extra training data)
Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training
Jointist consists of an instrument recognition module that conditions the other two modules: a transcription module that outputs instrument-specific piano rolls, and a source separation module that utilizes instrument information and transcription results.
Ontology-aware Learning and Evaluation for Audio Tagging
The proposed metric, ontology-aware mean average precision (OmAP) addresses the weaknesses of mAP by utilizing the AudioSet ontology information during the evaluation.
Binaural Rendering of Ambisonic Signals by Neural Networks
Binaural rendering of ambisonic signals is of broad interest to virtual reality and immersive media.
Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention
Audio captioning aims to generate text descriptions of audio clips.
Streaming Voice Conversion Via Intermediate Bottleneck Features And Non-streaming Teacher Guidance
In this paper, we propose to use intermediate bottleneck features (IBFs) to replace PPGs.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Learning Temporal Resolution in Spectrogram for Audio Classification
The audio spectrogram is a time-frequency representation that has been widely used for audio classification.
Simple Pooling Front-ends For Efficient Audio Classification
Recently, there has been increasing interest in building efficient audio neural networks for on-device scenarios.
Sound Event Localization and Detection for Real Spatial Sound Scenes: Event-Independent Network and Data Augmentation Chains
Our system submitted to the DCASE 2022 Task 3 is based on our previous proposed Event-Independent Network V2 (EINV2) with a novel data augmentation method.
Segment-level Metric Learning for Few-shot Bioacoustic Event Detection
In addition, we use transductive inference on the validation set during training for better adaptation to novel classes.
Jointist: Joint Learning for Multi-instrument Transcription and Its Applications
However, its novelty necessitates a new perspective on how to evaluate such a model.
Ranked #4 on
Music Transcription
on Slakh2100
VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration
Speech restoration aims to remove distortions in speech signals.
Neural Vocoder is All You Need for Speech Super-resolution
In this paper, we propose a neural vocoder based speech super-resolution method (NVSR) that can handle a variety of input resolution and upsampling ratios.
Ranked #2 on
Audio Super-Resolution
on VCTK Multi-Speaker
Separate What You Describe: Language-Queried Audio Source Separation
In this paper, we introduce the task of language-queried audio source separation (LASS), which aims to separate a target source from an audio mixture based on a natural language query of the target source (e. g., "a man tells a joke followed by people laughing").
CWS-PResUNet: Music Source Separation with Channel-wise Subband Phase-aware ResUNet
On the MUSDB18HQ test set, we propose a 276-layer CWS-PResUNet and achieve state-of-the-art (SoTA) performance on vocals with an 8. 92 signal-to-distortion ratio (SDR) score.
Ranked #11 on
Music Source Separation
on MUSDB18
A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis
We propose a unified model for three inter-related tasks: 1) to \textit{separate} individual sound sources from a mixed music audio, 2) to \textit{transcribe} each sound source to MIDI notes, and 3) to\textit{ synthesize} new pieces based on the timbre of separated sources.
Time-domain Speech Enhancement with Generative Adversarial Learning
Speech enhancement aims to obtain speech signals with high intelligibility and quality from noisy speech.
Large-Scale MIDI-based Composer Classification
Music classification is a task to classify a music piece into labels such as genres or composers.
An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection
Polyphonic sound event localization and detection (SELD), which jointly performs sound event detection (SED) and direction-of-arrival (DoA) estimation, detects the type and occurrence time of sound events as well as their corresponding DoA angles simultaneously.
Sound Audio and Speech Processing
GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music
In this article, we create a GiantMIDI-Piano (GP) dataset containing 38, 700, 838 transcribed notes and 10, 855 unique solo piano works composed by 2, 786 composers.
High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times
In addition, previous AMT systems are sensitive to the misaligned onset and offset labels of audio recordings.
Ranked #5 on
Music Transcription
on MAESTRO
Music Transcription
Sound
Audio and Speech Processing
Event-Independent Network for Polyphonic Sound Event Localization and Detection
In this paper, a novel event-independent network for polyphonic sound event localization and detection is proposed.
Audio and Speech Processing Sound
DD-CNN: Depthwise Disout Convolutional Neural Network for Low-complexity Acoustic Scene Classification
This paper presents a Depthwise Disout Convolutional Neural Network (DD-CNN) for the detection and classification of urban acoustic scenes.
Audio Tagging by Cross Filtering Noisy Labels
Yet, it is labor-intensive to accurately annotate large amount of audio data, and the dataset may contain noisy labels in the practical settings.
Learning with Out-of-Distribution Data for Audio Classification
The proposed method uses an auxiliary classifier, trained on data that is known to be in-distribution, for detection and relabelling.
Deep Learning Based Energy Disaggregation and On/Off Detection of Household Appliances
On the basis of energy disaggregation, we then investigate the performance of two deep-learning based frameworks for the task of on/off detection which aims at estimating whether an appliance is in operation or not.
Single-Channel Signal Separation and Deconvolution with Generative Adversarial Networks
Single-channel signal separation and deconvolution aims to separate and deconvolve individual sources from a single-channel mixture and is a challenging problem in which no prior knowledge of the mixing filters is available.
Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy
In this paper, it is experimentally shown that the training information of SED is able to contribute to the direction of arrival estimation (DOAE).
Sound Audio and Speech Processing
General audio tagging with ensembling convolutional neural network and statistical features
Audio tagging is challenging due to the limited size of data and noisy labels.
Audio Tagging With Connectionist Temporal Classification Model Using Sequential Labelled Data
To use the order information of sound events, we propose sequential labelled data (SLD), where both the presence or absence and the order information of sound events are known.
Sound Event Detection and Time-Frequency Segmentation from Weakly Labelled Data
Sound event detection (SED) aims to detect when and recognize what sound events happen in an audio clip.
Sound Audio and Speech Processing
Multi-level Attention Model for Weakly Supervised Audio Classification
The objective of audio classification is to predict the presence or absence of audio events in an audio clip.
A joint separation-classification model for sound event detection of weakly labelled data
First, we propose a separation mapping from the time-frequency (T-F) representation of an audio to the T-F segmentation masks of the audio events.
Sound Audio and Speech Processing
Audio Set classification with attention model: A probabilistic perspective
Then the classification of a bag is the expectation of the classification output of the instances in the bag with respect to the learned probability measure.
Sound Audio and Speech Processing
Large-scale weakly supervised audio classification using gated convolutional neural network
In this paper, we present a gated convolutional neural network and a temporal attention-based localization method for audio classification, which won the 1st place in the large-scale weakly supervised sound event detection task of Detection and Classification of Acoustic Scenes and Events (DCASE) 2017 challenge.
Sound Audio and Speech Processing
Attention and Localization based on a Deep Convolutional Recurrent Model for Weakly Supervised Audio Tagging
Audio tagging aims to perform multi-label classification on audio chunks and it is a newly proposed task in the Detection and Classification of Acoustic Scenes and Events 2016 (DCASE 2016) challenge.
Sound
Convolutional Gated Recurrent Neural Network Incorporating Spatial Features for Audio Tagging
In this paper, we propose to use a convolutional neural network (CNN) to extract robust features from mel-filter banks (MFBs), spectrograms or even raw waveforms for audio tagging.
A Joint Detection-Classification Model for Audio Tagging of Weakly Labelled Data
The labeling of an audio clip is often based on the audio events in the clip and no event level label is provided to the user.
Sound
