Search Results for author:
Found 74 papers, 46 papers with code
WavCraft: Audio Editing and Generation with Natural Language Prompts
We introduce WavCraft, a collective system that leverages large language models (LLMs) to connect diverse task-specific models for audio content creation and editing.
Multi-level graph learning for audio event classification and human-perceived annoyance rating prediction
Specifically, this paper proposes a lightweight multi-level graph learning (MLGL) based on local and global semantic graphs to simultaneously perform audio event classification (AEC) and human annoyance rating prediction (ARP).
Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection
Sound event localization and detection (SELD) combines two subtasks: sound event detection (SED) and direction of arrival (DOA) estimation.
Acoustic Prompt Tuning: Empowering Large Language Models with Audition Capabilities
Moreover, we improve the framework of audio language model by using interleaved audio-text embeddings as the input sequence.
CM-PIE: Cross-modal perception for interactive-enhanced audio-visual video parsing
Audio-visual video parsing is the task of categorizing a video at the segment level with weak labels, and predicting them as audible or visible events.
Audio Event-Relational Graph Representation Learning for Acoustic Scene Classification
The results show the feasibility of recognizing diverse acoustic scenes based on the audio event-relational graph.
Acoustic Scene Classification
Graph Representation Learning
+1
Hierarchical Metadata Information Constrained Self-Supervised Learning for Anomalous Sound Detection Under Domain Shift
Self-supervised learning methods have achieved promising performance for anomalous sound detection (ASD) under domain shift, where the type of domain shift is considered in feature learning by incorporating section IDs.
Retrieval-Augmented Text-to-Audio Generation
Despite recent progress in text-to-audio (TTA) generation, we show that the state-of-the-art models, such as AudioLDM, trained on datasets with an imbalanced class distribution, such as AudioCaps, are biased in their generation performance.
Ranked #2 on
Audio Generation
on AudioCaps
AudioSR: Versatile Audio Super-resolution at Scale
Audio super-resolution is a fundamental task that predicts high-frequency components for low-resolution audio, enhancing audio quality in digital applications.
Joint Prediction of Audio Event and Annoyance Rating in an Urban Soundscape by Hierarchical Graph Representation Learning
Sound events in daily life carry rich information about the objective world.
META-SELD: Meta-Learning for Fast Adaptation to the new environment in Sound Event Localization and Detection
For learning-based sound event localization and detection (SELD) methods, different acoustic environments in the training and test sets may result in large performance differences in the validation and evaluation stages.
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
Any audio can be translated into LOA based on AudioMAE, a self-supervised pre-trained representation learning model.
Ranked #3 on
Audio Generation
on AudioCaps
Separate Anything You Describe
In this work, we introduce AudioSep, a foundation model for open-domain audio source separation with natural language queries.
WavJourney: Compositional Audio Creation with Large Language Models
Subjective evaluations demonstrate the potential of WavJourney in crafting engaging storytelling audio content from text.
Text-Driven Foley Sound Generation With Latent Diffusion Model
We have observed that the feature embedding extracted by the text encoder can significantly affect the performance of the generation model.
Knowledge Distillation for Efficient Audio-Visual Video Captioning
Automatically describing audio-visual content with texts, namely video captioning, has received significant attention due to its potential applications across diverse fields.
Dual Transformer Decoder based Features Fusion Network for Automated Audio Captioning
Experimental results show that LHDFF outperforms existing audio captioning models.
Adapting Language-Audio Models as Few-Shot Audio Learners
We presented the Treff adapter, a training-efficient adapter for CLAP, to boost zero-shot classification performance by making use of a small set of labelled data.
Time-weighted Frequency Domain Audio Representation with GMM Estimator for Anomalous Sound Detection
This paper presents Time-Weighted Frequency Domain Representation (TWFR) with the GMM method (TWFR-GMM) for anomalous sound detection.
Anomalous Sound Detection using Audio Representation with Machine ID based Contrastive Learning Pretraining
This paper uses contrastive learning to refine audio representations for each machine ID, rather than for each audio sample.
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions.
Ranked #1 on
Zero-Shot Environment Sound Classification
on ESC-50
(using extra training data)
Leveraging Pre-trained AudioLDM for Text to Sound Generation: A Benchmark Study
Deep neural networks have recently achieved breakthroughs in sound generation with text prompts.
Differentiable Bootstrap Particle Filters for Regime-Switching Models
Differentiable particle filters are an emerging class of particle filtering methods that use neural networks to construct and learn parametric state-space models.
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency.
Ranked #9 on
Audio Generation
on AudioCaps
Unpaired Overwater Image Defogging Using Prior Map Guided CycleGAN
In this work, we propose a Prior map Guided CycleGAN (PG-CycleGAN) for defogging of images with overwater scenes.
Towards Generating Diverse Audio Captions via Adversarial Training
Captions generated by existing models are generally faithful to the content of audio clips, however, these machine-generated captions are often deterministic (e. g., generating a fixed caption for a given audio clip), simple (e. g., using common words and simple grammar), and generic (e. g., generating the same caption for similar audio clips).
ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification
Transformers, which were originally developed for natural language processing, have recently generated significant interest in the computer vision and audio communities due to their flexibility in learning long-range relationships.
Ontology-aware Learning and Evaluation for Audio Tagging
The proposed metric, ontology-aware mean average precision (OmAP) addresses the weaknesses of mAP by utilizing the AudioSet ontology information during the evaluation.
Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention
Audio captioning aims to generate text descriptions of audio clips.
Multi-dimensional Edge-based Audio Event Relational Graph Representation Learning for Acoustic Scene Classification
Experiments on a polyphonic acoustic scene dataset show that the proposed ERGL achieves competitive performance on ASC by using only a limited number of embeddings of audio events without any data augmentations.
Acoustic Scene Classification
Graph Representation Learning
+1
Personalized Dialogue Generation with Persona-Adaptive Attention
Persona-based dialogue systems aim to generate consistent responses based on historical context and predefined persona.
Automated Audio Captioning via Fusion of Low- and High- Dimensional Features
Moreover, in LHDFF, a new PANNs encoder is proposed called Residual PANNs (RPANNs) by fusing the low-dimensional feature from the intermediate convolution layer output and the high-dimensional feature from the final layer output of PANNs.
Learning Temporal Resolution in Spectrogram for Audio Classification
The audio spectrogram is a time-frequency representation that has been widely used for audio classification.
Simple Pooling Front-ends For Efficient Audio Classification
Recently, there has been increasing interest in building efficient audio neural networks for on-device scenarios.
Low-complexity CNNs for Acoustic Scene Classification
This technical report describes the SurreyAudioTeam22s submission for DCASE 2022 ASC Task 1, Low-Complexity Acoustic Scene Classification (ASC).
Continual Learning For On-Device Environmental Sound Classification
Experimental results on the DCASE 2019 Task 1 and ESC-50 dataset show that our proposed method outperforms baseline continual learning methods on classification accuracy and computational efficiency, indicating our method can efficiently and incrementally learn new classes without the catastrophic forgetting problem for on-device environmental sound classification.
Segment-level Metric Learning for Few-shot Bioacoustic Event Detection
In addition, we use transductive inference on the validation set during training for better adaptation to novel classes.
Automated Audio Captioning: An Overview of Recent Progress and New Challenges
In this paper, we present a comprehensive review of the published contributions in automated audio captioning, from a variety of existing approaches to evaluation metrics and datasets.
On Metric Learning for Audio-Text Cross-Modal Retrieval
We present an extensive evaluation of popular metric learning objectives on the AudioCaps and Clotho datasets.
Separate What You Describe: Language-Queried Audio Source Separation
In this paper, we introduce the task of language-queried audio source separation (LASS), which aims to separate a target source from an audio mixture based on a natural language query of the target source (e. g., "a man tells a joke followed by people laughing").
Deep Neural Decision Forest for Acoustic Scene Classification
In this paper, we propose a novel approach for ASC using deep neural decision forest (DNDF).
Leveraging Pre-trained BERT for Audio Captioning
BERT is a pre-trained language model that has been extensively used in Natural Language Processing (NLP) tasks.
Local Information Assisted Attention-free Decoder for Audio Captioning
Although this method effectively captures global information within audio data via the self-attention mechanism, it may ignore the event with short time duration, due to its limitation in capturing local information in an audio signal, leading to inaccurate prediction of captions.
Diverse Audio Captioning via Adversarial Training
As different people may describe an audio clip from different aspects using distinct words and grammars, we argue that an audio captioning system should have the ability to generate diverse captions for a fixed audio clip and across similar audio clips.
End-to-end translation of human neural activity to speech with a dual-dual generative adversarial network
A potential approach to this problem is to design an end-to-end method by using a dual generative adversarial network (DualGAN) without dimension reduction of passing information, but it cannot realize one-to-one signal-to-signal translation (see Fig. 1 (a) and (b)).
One to Multiple Mapping Dual Learning: Learning Multiple Sources from One Mixed Signal
Single channel blind source separation (SCBSS) refers to separate multiple sources from a mixed signal collected by a single sensor.
ARCA23K: An audio dataset for investigating open-set label noise
We show that the majority of labelling errors in ARCA23K are due to out-of-vocabulary audio clips, and we refer to this type of label noise as open-set label noise.
An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning
Automated audio captioning aims to use natural language to describe the content of audio data.
Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning
We evaluate our approach on the UrbanSound8K dataset, compared to SampleRNN, with the performance metrics measuring the quality and diversity of generated sounds.
Audio Captioning Transformer
In this paper, we propose an Audio Captioning Transformer (ACT), which is a full Transformer network based on an encoder-decoder architecture and is totally convolution-free.
Ranked #8 on
Audio captioning
on AudioCaps
CL4AC: A Contrastive Loss for Audio Captioning
Automated Audio captioning (AAC) is a cross-modal translation task that aims to use natural language to describe the content of an audio clip.
Low-dimensional Denoising Embedding Transformer for ECG Classification
However, the high-dimensional embedding obtained via 1-D convolution and positional encoding can lead to the loss of the signal's own temporal information and a large amount of training parameters.
SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classification
In this paper, we present SpecAugment++, a novel data augmentation method for deep neural networks based acoustic scene classification (ASC).
Time-domain Speech Enhancement with Generative Adversarial Learning
Speech enhancement aims to obtain speech signals with high intelligibility and quality from noisy speech.
Enhancing Audio Augmentation Methods with Consistency Learning
For tasks such as classification, there is a good case for learning representations of the data that are invariant to such transformations, yet this is not explicitly enforced by classification losses such as the cross-entropy loss.
An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection
Polyphonic sound event localization and detection (SELD), which jointly performs sound event detection (SED) and direction-of-arrival (DoA) estimation, detects the type and occurrence time of sound events as well as their corresponding DoA angles simultaneously.
Sound Audio and Speech Processing
Event-Independent Network for Polyphonic Sound Event Localization and Detection
In this paper, a novel event-independent network for polyphonic sound event localization and detection is proposed.
Audio and Speech Processing Sound
Evolving Multi-Resolution Pooling CNN for Monaural Singing Voice Separation
Monaural Singing Voice Separation (MSVS) is a challenging task and has been studied for decades.
Learning with Out-of-Distribution Data for Audio Classification
The proposed method uses an auxiliary classifier, trained on data that is known to be in-distribution, for detection and relabelling.
Environmental Sound Classification with Parallel Temporal-spectral Attention
Convolutional neural networks (CNN) are one of the best-performing neural network architectures for environmental sound classification (ESC).
Acoustic Scene Classification
Environmental Sound Classification
+3
IENet: Interacting Embranchment One Stage Anchor Free Detector for Orientation Aerial Object Detection
First, a novel geometric transformation is employed to better represent the oriented object in angle prediction, then a branch interactive module with a self-attention mechanism is developed to fuse features from classification and box regression branches.
Single-Channel Signal Separation and Deconvolution with Generative Adversarial Networks
Single-channel signal separation and deconvolution aims to separate and deconvolve individual sources from a single-channel mixture and is a challenging problem in which no prior knowledge of the mixing filters is available.
Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy
In this paper, it is experimentally shown that the training information of SED is able to contribute to the direction of arrival estimation (DOAE).
Sound Audio and Speech Processing
Bayesian inference for PCA and MUSIC algorithms with unknown number of sources
We then use Bayesian method to, for the first time, compute the MAP estimate for the number of sources in PCA and MUSIC algorithms.
Sound Event Detection and Time-Frequency Segmentation from Weakly Labelled Data
Sound event detection (SED) aims to detect when and recognize what sound events happen in an audio clip.
Sound Audio and Speech Processing
A joint separation-classification model for sound event detection of weakly labelled data
First, we propose a separation mapping from the time-frequency (T-F) representation of an audio to the T-F segmentation masks of the audio events.
Sound Audio and Speech Processing
Audio Set classification with attention model: A probabilistic perspective
Then the classification of a bag is the expectation of the classification output of the instances in the bag with respect to the learned probability measure.
Sound Audio and Speech Processing
Large-scale weakly supervised audio classification using gated convolutional neural network
In this paper, we present a gated convolutional neural network and a temporal attention-based localization method for audio classification, which won the 1st place in the large-scale weakly supervised sound event detection task of Detection and Classification of Acoustic Scenes and Events (DCASE) 2017 challenge.
Sound Audio and Speech Processing
Attention and Localization based on a Deep Convolutional Recurrent Model for Weakly Supervised Audio Tagging
Audio tagging aims to perform multi-label classification on audio chunks and it is a newly proposed task in the Detection and Classification of Acoustic Scenes and Events 2016 (DCASE 2016) challenge.
Sound
Convolutional Gated Recurrent Neural Network Incorporating Spatial Features for Audio Tagging
In this paper, we propose to use a convolutional neural network (CNN) to extract robust features from mel-filter banks (MFBs), spectrograms or even raw waveforms for audio tagging.
A Joint Detection-Classification Model for Audio Tagging of Weakly Labelled Data
The labeling of an audio clip is often based on the audio events in the clip and no event level label is provided to the user.
Sound
Hierarchical learning for DNN-based acoustic scene classification
In this paper, we present a deep neural network (DNN)-based acoustic scene classification framework.
Unsupervised Feature Learning Based on Deep Models for Environmental Audio Tagging
For the unsupervised feature learning, we propose to use a symmetric or asymmetric deep de-noising auto-encoder (sDAE or aDAE) to generate new data-driven features from the Mel-Filter Banks (MFBs) features.
Fully DNN-based Multi-label regression for audio tagging
Compared with the conventional Gaussian Mixture Model (GMM) and support vector machine (SVM) methods, the proposed fully DNN-based method could well utilize the long-term temporal information with the whole chunk as the input.
