Search Results for author:
Found 39 papers, 21 papers with code
CoT-ST: Enhancing LLM-based Speech Translation with Multimodal Chain-of-Thought
We propose CoT-ST, a speech translation model that utilizes multimodal CoT to decompose speech translation into sequential steps of speech recognition and translation.
NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization
To address this issue, we propose a novel VQ method, Normal Distribution-based Vector Quantization (NDVQ), by introducing an explicit margin between the VQ codes via learning a variance.
Progressive Residual Extraction based Pre-training for Speech Representation Learning
In this way, we can progressively extract pitch variation, speaker, and content representations from the input speech.
Foundation Models for Music: A Survey
In recent years, foundation models (FMs) such as large language models (LLMs) and latent diffusion models (LDMs) have profoundly impacted diverse sectors, including music.
CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis.
FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs).
TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizers
Neural codec language model (LM) has demonstrated strong capability in zero-shot text-to-speech (TTS) synthesis.
GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement
Notably, ASR models trained on GigaSpeech 2 can reduce the word error rate for Thai, Indonesian, and Vietnamese on our challenging and realistic YouTube test set by 25% to 40% compared to the Whisper large-v3 model, with merely 10% model parameters.
EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark
In this paper, we propose EmoBox, an out-of-the-box multilingual multi-corpus speech emotion recognition toolkit, along with a benchmark for both intra-corpus and cross-corpus settings.
MaLa-ASR: Multimedia-Assisted LLM-Based ASR
As more and more information-rich data like video become available, utilizing multi-modal auxiliary information to enhance audio tasks has sparked widespread research interest.
LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR
Recent years have witnessed significant progress in multilingual automatic speech recognition (ASR), driven by the emergence of end-to-end (E2E) models and the scaling of multilingual datasets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
1st Place Solution to Odyssey Emotion Recognition Challenge Task1: Tackling Class Imbalance Problem
Previous work has utilised class weighted loss for training, but problems remain as it sometimes causes over-fitting for minor classes or under-fitting for major classes.
MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series
To improve the transparency of LLMs, the research community has formed to open-source truly open LLMs (e. g., Pythia, Amber, OLMo), where more details (e. g., pre-training corpus and training code) are being provided.
MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition
However, this process may lead to inaccurate annotations, such as ignoring non-majority or non-candidate labels.
The X-LANCE Technical Report for Interspeech 2024 Speech Processing Using Discrete Speech Unit Challenge
Discrete speech tokens have been more and more popular in multiple speech processing fields, including automatic speech recognition (ASR), text-to-speech (TTS) and singing voice synthesis (SVS).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
MuPT: A Generative Symbolic Music Pretrained Transformer
In this paper, we explore the application of Large Language Models (LLMs) to the pre-training of music.
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs.
ChatMusician: Understanding and Generating Music Intrinsically with LLM
It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language.
An Embarrassingly Simple Approach for LLM with Strong ASR Capacity
We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
BAT: Learning to Reason about Spatial Sounds with Large Language Models
By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment.
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering
The language model (LM) approach based on acoustic and linguistic prompts, such as VALL-E, has achieved remarkable progress in the field of zero-shot audio generation.
EAT: Self-Supervised Pre-Training with Efficient Audio Transformer
Audio self-supervised learning (SSL) pre-training, which aims to learn good representations from unlabeled audio, has made remarkable progress.
Ranked #15 on
Audio Classification
on AudioSet
emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
To the best of our knowledge, emotion2vec is the first universal representation model in various emotion-related tasks, filling a gap in the field.
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT
Previous mainstream audio-and-text LLMs use discrete audio tokens to represent both input and output audio; however, they suffer from performance degradation on tasks such as automatic speech recognition, speech-to-text translation, and speech enhancement over models using continuous speech features.
Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning
Recent years have witnessed significant advancements in self-supervised learning (SSL) methods for speech-processing tasks.
Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition
In this paper, we explored how to boost speech emotion recognition (SER) with the state-of-the-art speech pre-trained model (PTM), data2vec, text generation technique, GPT-4, and speech synthesis technique, Azure TTS.
Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS
Self-supervised learning (SSL) proficiency in speech-related tasks has driven research into utilizing discrete tokens for speech tasks like recognition and translation, which offer lower storage requirements and great potential to employ natural language processing techniques.
VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching
Although diffusion models in text-to-speech have become a popular choice due to their strong generative ability, the intrinsic complexity of sampling from diffusion models harms their efficiency.
Unsupervised Active Learning: Optimizing Labeling Cost-Effectiveness for Automatic Speech Recognition
In recent years, speech-based self-supervised learning (SSL) has made significant progress in various tasks, including automatic speech recognition (ASR).
Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation
Our models outperform other SSL models significantly on the LibriSpeech benchmark without the need for iterative re-clustering and re-training.
Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentation
Recently, end-to-end (E2E) automatic speech recognition (ASR) models have made great strides and exhibit excellent performance in general speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
LTCR: Long-Text Chinese Rumor Detection Dataset
False information can spread quickly on social media, negatively influencing the citizens' behaviors and responses to social events.
Front-End Adapter: Adapting Front-End Input of Speech based Self-Supervised Learning for Speech Recognition
However, the training of SSL models is computationally expensive and a common practice is to fine-tune a released SSL model on the specific task.
MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets
In this paper, we provide a new perspective on self-supervised speech models from how the training targets are obtained.
Ranked #43 on
Speech Recognition
on LibriSpeech test-other
Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition
Recent years have witnessed great strides in self-supervised learning (SSL) on the speech processing.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Hierarchical Deep Residual Reasoning for Temporal Moment Localization
Temporal Moment Localization (TML) in untrimmed videos is a challenging task in the field of multimedia, which aims at localizing the start and end points of the activity in the video, described by a sentence query.
Video Super-Resolution via Deep Draft-Ensemble Learning
We propose a new direction for fast video super-resolution (VideoSR) via a SR draft ensemble, which is defined as the set of high-resolution patch candidates before final image deconvolution.
Handling Motion Blur in Multi-Frame Super-Resolution
Ubiquitous motion blur easily fails multi-frame super-resolution (MFSR).
Bounded-Distortion Metric Learning
Metric learning aims to embed one metric space into another to benefit tasks like classification and clustering.
