Search Results for author:
Found 38 papers, 8 papers with code
Characteristic-Specific Partial Fine-Tuning for Efficient Emotion and Speaker Adaptation in Codec Language Text-to-Speech Models
To address these challenges, we propose a characteristic-specific partial fine-tuning strategy, short as CSP-FT. First, we use a weighted-sum approach to analyze the contributions of different Transformer layers in a pre-trained codec language TTS model for emotion and speaker control in the generated speech.
Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding
In this paper, we adapt Whisper, which is a large-scale multilingual pre-trained speech recognition model, to CS from both encoder and decoder parts.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement
Experimental results on the VCTK+DEMAND dataset indicate that Mamba-SEUNet attains a PESQ score of 3. 59, while maintaining low computational complexity.
Ranked #2 on
Speech Enhancement
on VoiceBank + DEMAND
Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content
Multimodal Sentiment Analysis (MSA) stands as a critical research frontier, seeking to comprehensively unravel human emotions by amalgamating text, audio, and visual data.
Progressive Residual Extraction based Pre-training for Speech Representation Learning
In this way, we can progressively extract pitch variation, speaker, and content representations from the input speech.
VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing
For tasks such as text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), a cross-modal fine-grained (frame-level) sequence representation is desired, emphasizing the semantic content of the text modality while de-emphasizing the paralinguistic information of the speech modality.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition
Accurately finding the wrong words in the automatic speech recognition (ASR) hypothesis and recovering them well-founded is the goal of speech error correction.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios
Self-supervised learning (SSL) representations from massively multilingual models offer a promising solution for low-resource language speech tasks.
AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations
On the other hand, the Modality Interaction Network performs interaction fusion of extracted inter-modal features and intra-modal features.
ICMC-ASR: The ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition Challenge
To promote speech processing and recognition research in driving scenarios, we build on the success of the Intelligent Cockpit Speech Recognition Challenge (ICSRC) held at ISCSLP 2022 and launch the ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition (ICMC-ASR) Challenge.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
A Refining Underlying Information Framework for Monaural Speech Enhancement
By bridging the speech enhancement and the Information Bottleneck principle in this letter, we rethink a universal plug-and-play strategy and propose a Refining Underlying Information framework called RUI to rise to the challenges both in theory and practice.
High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
To address these issues, we propose a minimally-supervised high-fidelity speech synthesis method, where all modules are constructed based on the diffusion models.
Learning Speech Representation From Contrastive Token-Acoustic Pretraining
However, existing contrastive learning methods in the audio field focus on extracting global descriptive information for downstream audio classification tasks, making them unsuitable for TTS, VC, and ASR tasks.
Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding
However, existing methods suffer from three problems: the high dimensionality and waveform distortion of discrete speech representations, the prosodic averaging problem caused by the duration prediction model in non-autoregressive frameworks, and the information redundancy and dimension explosion problems of existing semantic encoding methods.
Locate and Beamform: Two-dimensional Locating All-neural Beamformer for Multi-channel Speech Separation
Recently, stunning improvements on multi-channel speech separation have been achieved by neural beamformers when direction information is available.
Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification
Visual speech (i. e., lip motion) is highly related to auditory speech due to the co-occurrence and synchronization in speech production.
MIMO-DBnet: Multi-channel Input and Multiple Outputs DOA-aware Beamforming Network for Speech Separation
Recently, many deep learning based beamformers have been proposed for multi-channel speech separation.
I4U System Description for NIST SRE'20 CTS Challenge
This manuscript describes the I4U submission to the 2020 NIST Speaker Recognition Evaluation (SRE'20) Conversational Telephone Speech (CTS) Challenge.
Monolingual Recognizers Fusion for Code-switching Speech Recognition
It has two stages: the speech awareness (SA) stage and the language fusion (LF) stage.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Deep Spectro-temporal Artifacts for Detecting Synthesized Speech
The Audio Deep Synthesis Detection (ADD) Challenge has been held to detect generated human-like speech.
VCSE: Time-Domain Visual-Contextual Speaker Extraction Network
In the first stage, we pre-extract a target speech with visual cues and estimate the underlying phonetic sequence.
MIMO-DoAnet: Multi-channel Input and Multiple Outputs DoA Network with Unknown Number of Sound Sources
These algorithms are usually achieved by mapping the multi-channel audio input to the single output (i. e. overall spatial pseudo-spectrum (SPS) of all sources), that is called MISO.
Language-specific Characteristic Assistance for Code-switching Speech Recognition
Dual-encoder structure successfully utilizes two language-specific encoders (LSEs) for code-switching speech recognition.
Iterative Sound Source Localization for Unknown Number of Sources
Sound source localization aims to seek the direction of arrival (DOA) of all sound sources from the observed multi-channel audio.
Talking Head Generation Driven by Speech-Related Facial Action Units and Audio- Based on Multimodal Representation Fusion
We utilize pre-trained AU classifier to ensure that the generated images contain correct AU information.
L-SpEx: Localized Target Speaker Extraction
Speaker extraction aims to extract the target speaker's voice from a multi-talker speech mixture given an auxiliary reference utterance.
Talking Head Generation with Audio and Speech Related Facial Action Units
Quantitative and qualitative experiments demonstrate that our method outperforms existing methods in both image quality and lip-sync accuracy.
Using multiple reference audios and style embedding constraints for speech synthesis
The end-to-end speech synthesis model can directly take an utterance as reference audio, and generate speech from the text with prosody and speaker characteristics similar to the reference audio.
Exploring Deep Learning for Joint Audio-Visual Lip Biometrics
Audio-visual (AV) lip biometrics is a promising authentication technique that leverages the benefits of both the audio and visual modalities in speech communication.
Multi-stage Speaker Extraction with Utterance and Frame-Level Reference Signals
Speaker extraction requires a sample speech from the target speaker as the reference.
SpEx+: A Complete Time Domain Speaker Extraction Network
To eliminate such mismatch, we propose a complete time-domain speaker extraction solution, that is called SpEx+.
Ranked #1 on
Speech Extraction
on WSJ0-2mix-extr
Speech Extraction
Audio and Speech Processing
Sound
Towards Efficient Processing and Learning with Spikes: New Approaches for Multi-Spike Learning
They are also believed to play an essential role in low-power consumption of the biological systems, whose efficiency attracts increasing attentions to the field of neuromorphic computing.
A Semi-Supervised Stable Variational Network for Promoting Replier-Consistency in Dialogue Generation
However, the sampled information from latent space usually becomes useless due to the KL divergence vanishing issue, and the highly abstractive global variables easily dilute the personal features of replier, leading to a non replier-specific response.
Relation Modeling with Graph Convolutional Networks for Facial Action Unit Detection
Most existing AU detection works considering AU relationships are relying on probabilistic graphical models with manually extracted features.
Robust Environmental Sound Recognition with Sparse Key-point Encoding and Efficient Multi-spike Learning
Our framework is a unifying system with a consistent integration of three major functional parts which are sparse encoding, efficient learning and robust readout.
Implicit Discourse Relation Recognition using Neural Tensor Network with Interactive Attention and Sparse Learning
In this paper, we propose a novel neural Tensor network framework with Interactive Attention and Sparse Learning (TIASL) for implicit discourse relation recognition.
Interaction-Aware Topic Model for Microblog Conversations through Network Embedding and User Attention
They ignore that one discusses diverse topics when dynamically interacting with different people.
Speech Emotion Recognition Considering Local Dynamic Features
Recently, increasing attention has been directed to the study of the speech emotion recognition, in which global acoustic features of an utterance are mostly used to eliminate the content differences.
