Search Results for author:
Found 43 papers, 35 papers with code
SVDD 2024: The Inaugural Singing Voice Deepfake Detection Challenge
With the advancements in singing voice generation and the growing presence of AI singers on media platforms, the inaugural Singing Voice Deepfake Detection (SVDD) Challenge aims to advance research in identifying AI-generated singing voices from authentic singers.
Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition
In this work, we propose a synthetic data generation pipeline for multi-speaker conversational ASR, leveraging a large language model (LLM) for content creation and a conversational multi-speaker text-to-speech (TTS) model for speech synthesis.
A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection
Additionally, to ensure the credibility of detection methods, it is beneficial for the model to interpret which cues from the video indicate it is fake.
Articulatory Phonetics Informed Controllable Expressive Speech Synthesis
We open-source the dataset and TTS models.
CtrSVDD: A Benchmark Dataset and Baseline Analysis for Controlled Singing Voice Deepfake Detection
Addressing these gaps, we introduce CtrSVDD, a large-scale, diverse collection of bonafide and deepfake singing vocals.
SVDD Challenge 2024: A Singing Voice Deepfake Detection Challenge Evaluation Plan
The rapid advancement of AI-generated singing voices, which now closely mimic natural human singing and align seamlessly with musical scores, has led to heightened concerns for artists and the music industry.
Scoring Time Intervals using Non-Hierarchical Transformer For Automatic Piano Transcription
The neural semi-Markov Conditional Random Field (semi-CRF) framework has demonstrated promise for event-based piano transcription.
Ranked #1 on
Music Transcription
on SMD Piano
Toward Fully Self-Supervised Multi-Pitch Estimation
Multi-pitch estimation is a decades-long research problem involving the detection of pitch activity associated with concurrent musical events within multi-instrument mixtures.
Learning Arousal-Valence Representation from Categorical Emotion Labels of Speech
In this work, we propose to learn the AV representation from categorical emotion labels of speech.
SynthTab: Leveraging Synthesized Data for Guitar Tablature Transcription
Guitar Tablature Transcription (GTT) is an important task with broad applications in music education, composition, and entertainment.
SingFake: Singing Voice Deepfake Detection
These unique properties make singing voice deepfake detection a relevant but significantly different problem from synthetic speech detection.
Mitigating Cross-Database Differences for Learning Unified HRTF Representation
We further show that these normalized HRTFs can be used to learn a more unified HRTF representation across databases than the prior art.
SingNet: A Real-time Singing Voice Beat and Downbeat Tracking System
Singing voice beat and downbeat tracking posses several applications in automatic music production, analysis and manipulation.
Transcription free filler word detection with Neural semi-CRFs
Non-linguistic filler words, such as "uh" or "um", are prevalent in spontaneous speech and serve as indicators for expressing hesitation or uncertainty.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SAMO: Speaker Attractor Multi-Center One-Class Learning for Voice Anti-Spoofing
Our previous research on one-class learning has improved the generalization ability to unseen attacks by compacting the bona fide speech in the embedding space.
HRTF Field: Unifying Measured HRTF Magnitude Representation with Neural Fields
In this work, we propose to use neural fields, a differentiable representation of functions through neural networks, to model HRTFs with arbitrary spatial sampling schemes.
ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Speed
In this paper, we propose ControlVC, the first neural voice conversion system that achieves time-varying controls on pitch and speed.
Singing Beat Tracking With Self-supervised Front-end and Linear Transformers
Tracking beats of singing voices without the presence of musical accompaniment can find many applications in music production, automatic song arrangement, and social media interaction.
Predicting Global Head-Related Transfer Functions From Scanned Head Geometry Using Deep Learning and Compact Representations
For the HRTF data, we use truncated spherical harmonic (SH) coefficients to represent the HRTF magnitudes and onsets.
Rethinking Audio-visual Synchronization for Active Speaker Detection
This clarification of definition is motivated by our extensive experiments, through which we discover that existing ASD methods fail in modeling the audio-visual synchronization and often classify unsynchronized videos as active speaking.
Music Source Separation with Generative Flow
Fully-supervised models for source separation are trained on parallel mixture-source data and are currently state-of-the-art.
A Data-Driven Methodology for Considering Feasibility and Pairwise Likelihood in Deep Learning Based Guitar Tablature Transcription Systems
This naturally enforces playability constraints for guitar, and yields tablature which is more consistent with the symbolic data used to estimate pairwise likelihoods.
A Probabilistic Fusion Framework for Spoofing Aware Speaker Verification
We further propose fusion strategies for direct inference and fine-tuning to predict the SASV score based on the framework.
Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs
When formulating piano transcription in this way, we eliminate the need to rely on disjoint frame-level estimates for different stages of a note event.
Ranked #7 on
Music Transcription
on MAESTRO
A Novel 1D State Space for Efficient Music Rhythmic Analysis
Inferring music time structures has a broad range of applications in music production, processing and analysis.
Ranked #1 on
Online Beat Tracking
on GTZAN
A study of the robustness of raw waveform based speaker embeddings under mismatched conditions
In this paper, we conduct a cross-dataset study on parametric and non-parametric raw-waveform based speaker embeddings through speaker verification experiments.
Learning Sparse Analytic Filters for Piano Transcription
In this work, several variations of a frontend filterbank learning module are investigated for piano transcription, a challenging low-level music information retrieval task.
BeatNet: CRNN and Particle Filtering for Online Joint Beat Downbeat and Meter Tracking
The online estimation of rhythmic information, such as beat positions, downbeat positions, and meter, is critical for many real-time music applications.
Ranked #1 on
Online Beat Tracking
on Rock Corpus
UR Channel-Robust Synthetic Speech Detection System for ASVspoof 2021
Different from previous ASVspoof challenges, the LA task this year presents codec and transmission channel variability, while the new task DF presents general audio compression.
Audiovisual Singing Voice Separation
Separating a song into vocal and accompaniment components is an active research topic, and recent years witnessed an increased performance from supervised training using deep learning techniques.
An Empirical Study on Channel Effects for Synthetic Voice Spoofing Countermeasure Systems
Spoofing countermeasure (CM) systems are critical in speaker verification; they aim to discern spoofing attacks from bona fide speech trials.
When Counterpoint Meets Chinese Folk Melodies
An interaction reward model is trained on the duets formed from outer parts of Bach chorales to model counterpoint interaction, while a style reward model is trained on monophonic melodies of Chinese folk songs to model melodic patterns.
Don't look back: an online beat tracking method using RNN and enhanced particle filtering
Most preexisting OBT methods either apply some offline approaches to a moving window containing past data to make predictions about future beat positions or must be primed with past data at startup to initialize.
One-class learning towards generalized voice spoofing detection
Human voices can be used to authenticate the identity of the speaker, but the automatic speaker verification (ASV) systems are vulnerable to voice spoofing attacks, such as impersonation, replay, text-to-speech, and voice conversion.
Y-Vector: Multiscale Waveform Encoder for Speaker Embedding
State-of-the-art text-independent speaker verification systems typically use cepstral features or filter bank energies as speech features.
Themes Informed Audio-visual Correspondence Learning
The applications of short-term user-generated video (UGV), such as Snapchat, and Youtube short-term videos, booms recently, raising lots of multimodal machine learning tasks.
Speech Driven Talking Face Generation from a Single Image and an Emotion Condition
Visual emotion expression plays an important role in audiovisual speech communication.
RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning
We cast this as a reinforcement learning problem, where the generation agent learns a policy to generate a musical note (action) based on previously generated context (state).
Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss
We devise a cascade GAN approach to generate talking face video, which is robust to different face shapes, view angles, facial characteristics, and noisy audio conditions.
Lip Movements Generation at a Glance
In this paper, we consider a task of such: given an arbitrary audio speech and one lip image of arbitrary target identity, generate synthesized lip movements of the target identity saying the speech.
Generating Talking Face Landmarks from Speech
In this paper, we present a system that can generate landmark points of a talking face from an acoustic speech in real time.
Audio-Visual Event Localization in Unconstrained Videos
In this paper, we introduce a novel problem of audio-visual event localization in unconstrained videos.
Deep Cross-Modal Audio-Visual Generation
Being the first to explore this new problem, we compose two new datasets with pairs of images and sounds of musical performances of different instruments.
