Search Results for author:
Found 63 papers, 18 papers with code
Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis
One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from an unseen image, and then animate it with a reference video or audio to generate a talking portrait video.
Improving Large-scale Deep Biasing with Phoneme Features and Text-only Data in Streaming Transducer
Deep biasing for the Transducer can improve the recognition performance of rare words or contextual entities, which is essential in practical applications, especially for streaming Automatic Speech Recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general auditory information consisting of at least three types of sounds: speech, audio events, and music.
Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models
Audio-visual large language models (LLM) have drawn significant attention, yet the fine-grained combination of both input streams is rather under-explored, which is challenging but necessary for LLMs to understand general video inputs.
Connecting Speech Encoder and Large Language Model for ASR
Q-Former-based LLMs can generalise well to out-of-domain datasets, where 12% relative WER reductions over the Whisper baseline ASR model were achieved on the Eval2000 test set without using any in-domain training data from Switchboard.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis
However, the prompting mechanisms of zero-shot TTS still face challenges in the following aspects: 1) previous works of zero-shot TTS are typically trained with single-sentence prompts, which significantly restricts their performance when the data is relatively sufficient during the inference stage.
GenerTTS: Pronunciation Disentanglement for Timbre and Style Generalization in Cross-Lingual Text-to-Speech
Cross-lingual timbre and style generalizable text-to-speech (TTS) aims to synthesize speech with a specific reference timbre or style that is never trained in the target language.
Towards Building Voice-based Conversational Recommender Systems: Datasets, Potential Solutions, and Prospects
Conversational recommender systems (CRSs) have become crucial emerging research topics in the field of RSs, thanks to their natural advantages of explicitly acquiring user preferences via interactive conversations and revealing the reasons behind recommendations.
Improving Frame-level Classifier for Word Timings with Non-peaky CTC in End-to-End Automatic Speech Recognition
In this paper, we improve the frame-level classifier for word timings in E2E system by introducing label priors in connectionist temporal classification (CTC) loss, which is adopted from prior works, and combining low-level Mel-scale filter banks with high-level ASR encoder output as input feature.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Text-only Domain Adaptation using Unified Speech-Text Representation in Transducer
Domain adaptation using text-only corpus is challenging in end-to-end(E2E) speech recognition.
Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
We are interested in a novel task, namely low-resource text-to-talking avatar.
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias
3) We further use a VQGAN-based acoustic model to generate the spectrogram and a latent code language model to fit the distribution of prosody, since prosody changes quickly over time in a sentence, and language models can capture both local and long-range dependencies.
PolyVoice: Language Models for Speech to Speech Translation
For the speech synthesis part, we adopt the existing VALL-E X approach and build a unit-based audio language model.
Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
Finally, we use LLMs to augment and transform a large amount of audio-label data into audio-text datasets to alleviate the problem of scarcity of temporal data.
Ranked #7 on
Audio Generation
on AudioCaps
CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training
We also observe the cross-modal representation extracted by CIF-PT obtains better performance than other neural interfaces for the tasks of SLU, including the dominant speech representation learned from self-supervised pre-training.
Phonetic and Prosody-aware Self-supervised Learning Approach for Non-native Fluency Scoring
Specifically, we first pre-train the model using a reconstruction loss function, by masking phones and their durations jointly on a large amount of unlabeled speech and text prompts.
GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
Recently, neural radiance field (NeRF) has become a popular rendering technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video.
Enhancing Large Language Model with Self-Controlled Memory Framework
Large Language Models (LLMs) are constrained by their inability to process lengthy inputs, resulting in the loss of critical historical information.
ByteCover3: Accurate Cover Song Identification on Short Queries
Deep learning based methods have become a paradigm for cover song identification (CSI) in recent years, where the ByteCover systems have achieved state-of-the-art results on all the mainstream datasets of CSI.
LiteG2P: A fast, light and high accuracy model for grapheme-to-phoneme conversion
As a key component of automated speech recognition (ASR) and the front-end in text-to-speech (TTS), grapheme-to-phoneme (G2P) plays the role of converting letters to their corresponding pronunciations.
Leveraging phone-level linguistic-acoustic similarity for utterance-level pronunciation scoring
Recent studies on pronunciation scoring have explored the effect of introducing phone embeddings as reference pronunciation, but mostly in an implicit manner, i. e., addition or concatenation of reference phone embedding and actual pronunciation of the target phone as the phone-level pronunciation quality representation.
An ASR-free Fluency Scoring Approach with Self-Supervised Learning
A typical fluency scoring system generally relies on an automatic speech recognition (ASR) system to obtain time stamps in input speech for either the subsequent calculation of fluency-related features or directly modeling speech fluency with an end-to-end approach.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Virtual Try-On with Pose-Garment Keypoints Guided Inpainting
In our method, human pose and garment keypoints are extracted from source images and constructed as graphs to predict the garment keypoints at the target pose.
Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features
Speech-to-speech translation directly translates a speech utterance to another between different languages, and has great potential in tasks such as simultaneous interpretation.
BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance
We highlight that benefiting from the compact architecture and optimized hardware kernel, BiFSMNv2 can achieve an impressive 25. 1x speedup and 20. 2x storage-saving on edge hardware.
Graph Contrastive Learning with Implicit Augmentations
Existing graph contrastive learning methods rely on augmentation techniques based on random perturbations (e. g., randomly adding or dropping edges and nodes).
Internal Language Model Estimation based Adaptive Language Model Fusion for Domain Adaptation
The proposed method can achieve significantly better performance on the target test sets while it gets minimal performance degradation on the general test set, compared with both shallow and ILME-based LM fusion methods.
Random Utterance Concatenation Based Data Augmentation for Improving Short-video Speech Recognition
One of limitations in end-to-end automatic speech recognition (ASR) framework is its performance would be compromised if train-test utterance lengths are mismatched.
Importance Prioritized Policy Distillation
Given such an issue, we denote the \emph{frame importance} as its contribution to the expected reward on a particular frame, and hypothesize that adapting such frame importance could benefit the performance of the distilled student policy.
A Novel Chinese Dialect TTS Frontend with Non-Autoregressive Neural Machine Translation
In this paper, we propose a novel Chinese dialect TTS frontend with a translation module, which converts Mandarin text into dialectic expressions to improve the intelligibility and naturalness of synthesized speech.
Improving Contextual Representation with Gloss Regularized Pre-training
To tackle this issue, we propose an auxiliary gloss regularizer module to BERT pre-training (GR-BERT), to enhance word semantic similarity.
BYTECOVER2: TOWARDS DIMENSIONALITY REDUCTION OF LATENT EMBEDDING FOR EFFICIENT COVER SONG IDENTIFICATION
Convolutional neural network (CNN)-based methods have dominated the recent research of cover song identification (CSI).
Ranked #1 on
Cover song identification
on SHS100K-TEST
Language Adaptive Cross-lingual Speech Representation Learning with Sparse Sharing Sub-networks
It makes room for language specific modeling by pruning out unimportant parameters for each language, without requiring any manually designed language specific component.
Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information
Moreover, we utilize multi-source information (e. g., MFCC and deep features) to further improve the scoring system performance.
S3T: Self-Supervised Pre-training with Swin Transformer for Music Classification
To our knowledge, S3T is the first method combining the Swin Transformer with a self-supervised learning method for music classification.
BiFSMN: Binary Neural Network for Keyword Spotting
Then, to allow the instant and adaptive accuracy-efficiency trade-offs at runtime, we also propose a Thinnable Binarization Architecture to further liberate the acceleration potential of the binarized network from the topology perspective.
The Volcspeech system for the ICASSP 2022 multi-channel multi-party meeting transcription challenge
For Track 1, we propose several approaches to empower the clustering-based speaker diarization system to handle overlapped speech.
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection
To combat these problems, we introduce HTS-AT: an audio transformer with a hierarchical structure to reduce the model size and training time.
Ranked #4 on
Sound Event Detection
on DESED
Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection
Nowadays, most methods in end-to-end contextual speech recognition bias the recognition process towards contextual knowledge.
Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASR
In this paper we propose two novel approaches to estimate the ILM based on Listen-Attend-Spell (LAS) framework.
Towards Realistic Visual Dubbing with Heterogeneous Sources
The task of few-shot visual dubbing focuses on synchronizing the lip movements with arbitrary speech input for any talking head video.
Zero-shot Audio Source Separation through Query-based Learningfrom Weakly-labeled Data
Our approach uses a single model for source separation of multiple sound types, and relies solely on weakly-labeled data for training.
Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Data
Our approach uses a single model for source separation of multiple sound types, and relies solely on weakly-labeled data for training.
Ranked #1 on
Audio Source Separation
on AudioSet
Towards Using Clothes Style Transfer for Scenario-aware Person Video Generation
To tackle this problem, most recent AdaIN-based architectures are proposed to extract clothes and scenario features for generation.
Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding
Experiments show that, compared with the baseline models, our proposed model can significantly improve the naturalness of converted singing voices and the similarity with the target singer.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech
In expressive speech synthesis, there are high requirements for emotion interpretation.
Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Mask
In the recent trend of semi-supervised speech recognition, both self-supervised representation learning and pseudo-labeling have shown promising results.
Synthesising Audio Adversarial Examples for Automatic Speech Recognition
Adversarial examples in automatic speech recognition (ASR) are naturally sounded by humans yet capable of fooling well trained ASR models to transcribe incorrectly.
HMM-Free Encoder Pre-Training for Streaming RNN Transducer
This work describes an encoder pre-training procedure using frame-wise label to improve the training of streaming recurrent neural network transducer (RNN-T) model.
An Improved Transfer Model: Randomized Transferable Machine
Existing studies usually assume that the learned new feature representation is \emph{domain-invariant}, and thus train a transfer model $\mathcal{M}$ on the source domain.
Dynamic latency speech recognition with asynchronous revision
In this work we propose an inference technique, asynchronous revision, to unify streaming and non-streaming speech recognition models.
Improving RNN transducer with normalized jointer network
In this work, we analyze the cause of the huge gradient variance in RNN-T training and proposed a new \textit{normalized jointer network} to overcome it.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
PPG-based singing voice conversion with adversarial representation learning
Singing voice conversion (SVC) aims to convert the voice of one singer to that of other singers while keeping the singing content and melody.
ByteCover: Cover Song Identification via Multi-Loss Training
We present in this paper ByteCover, which is a new feature learning method for cover song identification (CSI).
Ranked #2 on
Cover song identification
on Da-TACOS
Rule-embedded network for audio-visual voice activity detection in live musical video streams
Detecting anchor's voice in live musical streams is an important preprocessing for music and speech signal processing.
Sound Multimedia Audio and Speech Processing
Contrastive Unsupervised Learning for Audio Fingerprinting
The rise of video-sharing platforms has attracted more and more people to shoot videos and upload them to the Internet.
Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech
Accent conversion (AC) transforms a non-native speaker's accent into a native accent while maintaining the speaker's voice timbre.
ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders
This paper presents ByteSing, a Chinese singing voice synthesis (SVS) system based on duration allocated Tacotron-like acoustic models and WaveRNN neural vocoders.
A hybrid text normalization system using multi-head self-attention for mandarin
In this paper, we propose a hybrid text normalization system using multi-head self-attention.
A unified sequence-to-sequence front-end model for Mandarin text-to-speech synthesis
In Mandarin text-to-speech (TTS) system, the front-end text processing module significantly influences the intelligibility and naturalness of synthesized speech.
Frame Stacking and Retaining for Recurrent Neural Network Acoustic Model
The system which combined frame retaining with frame stacking could reduces the time consumption of both training and decoding.
Deep LSTM for Large Vocabulary Continuous Speech Recognition
It is a competitive framework that LSTM models of more than 7 layers are successfully trained on Shenma voice search data in Mandarin and they outperform the deep LSTM models trained by conventional approach.
Exponential Moving Average Model in Parallel Speech Recognition Training
As training data rapid growth, large-scale parallel training with multi-GPUs cluster is widely applied in the neural network model learning currently. We present a new approach that applies exponential moving average method in large-scale parallel training of neural network model.
