Search Results for author:
Found 40 papers, 22 papers with code
OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup
The scaling up has brought tremendous success in the fields of vision and language in recent years.
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
However, achieving low latency and natural interactions in full-duplex dialogue systems remains a significant challenge, especially considering human conversation dynamics such as interruptions, backchannels, and overlapping speech.
MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization
Generating music that aligns with the visual content of a video has been a challenging task, as it requires a deep understanding of visual semantics and involves generating music whose melody, rhythm, and dynamics harmonize with the visual narratives.
Exploring Text-Queried Sound Event Detection with Audio Source Separation
To address this issue, we propose the text-queried SED (TQ-SED) framework.
WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information.
Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization
Then we introduce a joint pairwise constraint propagation algorithm to cluster speakers based on these visual and semantic constraints.
CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis.
FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs).
Accompanied Singing Voice Synthesis with Fully Text-controlled Melody
Text-to-song (TTSong) is a music generation task that synthesizes accompanied singing voices.
Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
SDPN assigns the representation of the augmented views of an utterance to the same prototypes as the representation of the original view, thereby enabling effective knowledge transfer between the views.
Skip-Layer Attention: Bridging Abstract and Detailed Dependencies in Transformers
The Transformer architecture has significantly advanced deep learning, particularly in natural language processing, by effectively managing long-range dependencies.
ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Efficiency
Speaker verification systems experience significant performance degradation when tasked with short-duration trial recordings.
ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec
In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style, merely based on a few seconds of audio prompt and a simple textual style description prompt.
AudioLCM: Text-to-Audio Generation with Latent Consistency Models
To overcome the convergence issue inherent in LDMs with reduced sample iterations, we propose the Guided Latent Consistency Distillation with a multi-step Ordinary Differential Equation (ODE) solver.
3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization
With 3D-Speaker-Toolkit, we establish a new benchmark for multimodal speaker analysis.
Language-Codec: Reducing the Gaps Between Discrete Codec Representation and Speech Language Models
Furthermore, we also validate the efficiency of the Language-Codec on downstream speech language models.
An Embarrassingly Simple Approach for LLM with Strong ASR Capacity
We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token-based ASR
We find that applying the conventional cross-entropy loss on input speech tokens does not consistently improve the ASR performance over the Loss Masking approach.
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT
Previous mainstream audio-and-text LLMs use discrete audio tokens to represent both input and output audio; however, they suffer from performance degradation on tasks such as automatic speech recognition, speech-to-text translation, and speech enhancement over models using continuous speech features.
Improving Speaker Diarization using Semantic Information: Joint Pairwise Constraints Propagation
Speaker diarization has gained considerable attention within speech processing research community.
FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec
We also demonstrate that the pre-trained models are suitable for downstream tasks, including automatic speech recognition and personalized text-to-speech synthesis.
Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
To alleviate this problem, we introduce a diversity regularization term to embeddings in SDPN.
Improving BERT with Hybrid Pooling Network and Drop Mask
Transformer-based pre-trained language models, such as BERT, achieve great success in various natural language understanding tasks.
3D-Speaker: A Large-Scale Multi-Device, Multi-Distance, and Multi-Dialect Corpus for Speech Representation Disentanglement
Disentangling uncorrelated information in speech utterances is a crucial research topic within speech community.
Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization
In this paper, we propose methods to extract speaker-related information from semantic content in multi-party meetings, which, as we will show, can further benefit speaker diarization.
An Enhanced Res2Net with Local and Global Feature Fusion for Speaker Verification
This paper proposes a novel architecture called Enhanced Res2Net (ERes2Net), which incorporates both local and global feature fusion techniques to improve the performance.
Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings
Prior studies diagnose the anisotropy problem in sentence representations from pre-trained language models, e. g., BERT, without fine-tuning.
DopplerBAS: Binaural Audio Synthesis Addressing Doppler Effect
Recently, binaural audio synthesis (BAS) has emerged as a promising research field for its applications in augmented and virtual realities.
Contextual Expressive Text-to-Speech
To achieve this task, we construct a synthetic dataset and develop an effective framework.
Pushing the limits of self-supervised speaker verification using regularized distillation framework
A range of experiments conducted on the VoxCeleb datasets demonstrate the superiority of the regularized DINO framework in speaker verification.
Deep Representation Decomposition for Rate-Invariant Speaker Verification
While promising performance for speaker verification has been achieved by deep speaker embeddings, the advantage would reduce in the case of speaking-style variability.
Reformulating Speaker Diarization as Community Detection With Emphasis On Topological Structure
In this paper we propose to view clustering-based diarization as a community detection problem.
Graph Convolutional Network Based Semi-Supervised Learning on Multi-Speaker Meeting Data
In this work, we present a GCN-based approach for semi-supervised learning.
Speaker Embedding-aware Neural Diarization: an Efficient Framework for Overlapping Speech Diarization in Meeting Scenarios
Through this formulation, we propose the speaker embedding-aware neural diarization (SEND) framework, where a speech encoder, a speaker encoder, two similarity scorers, and a post-processing network are jointly optimized to predict the encoded labels according to the similarities between speech features and speaker embeddings.
Ranked #1 on
Speaker Diarization
on AliMeeting
Speaker Embedding-aware Neural Diarization for Flexible Number of Speakers with Textual Information
In this paper, we reformulate this task as a single-label prediction problem by encoding the multi-speaker labels with power set.
PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
We propose a novel Pooling Network (PoNet) for token mixing in long sequences with linear complexity.
BeamTransformer: Microphone Array-based Overlapping Speech Detection
We propose BeamTransformer, an efficient architecture to leverage beamformer's edge in spatial filtering and transformer's capability in context sequence modeling.
Measuring daily-life fear perception change: a computational study in the context of COVID-19
COVID-19, as a global health crisis, has triggered the fear emotion with unprecedented intensity.
A Real-time Speaker Diarization System Based on Spatial Spectrum
In this paper we describe a speaker diarization system that enables localization and identification of all speakers present in a conversation or meeting.
Estimating air quality co-benefits of energy transition using machine learning
Applications of the framework with Chinese data reveal highly heterogeneous health benefits of reducing fossil fuel use in different sectors and regions in China with a mean of \$34/tCO2 and a standard deviation of \$84/tCO2.
