Search Results for author:
Found 172 papers, 42 papers with code
NTU-NPU System for Voice Privacy 2024 Challenge
In this work, we describe our submissions for the Voice Privacy Challenge 2024.
Takin-VC: Zero-shot Voice Conversion via Jointly Hybrid Content and Memory-Augmented Context-Aware Timbre Modeling
Zero-shot voice conversion (VC) aims to transform the source speaker timbre into an arbitrary unseen one without altering the original speech content. While recent advancements in zero-shot VC methods have shown remarkable progress, there still remains considerable potential for improvement in terms of improving speaker similarity and speech naturalness. In this paper, we propose Takin-VC, a novel zero-shot VC framework based on jointly hybrid content and memory-augmented context-aware timbre modeling to tackle this challenge.
Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text
Integrating audio encoders with LLMs through connectors has enabled these models to process and comprehend audio modalities, significantly enhancing speech-to-text tasks, including automatic speech recognition (ASR) and automatic speech translation (AST).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
DualSep: A Light-weight dual-encoder convolutional recurrent network for real-time in-car speech separation
Our proposed system supports both streaming and non-streaming modes.
Findings of the 2024 Mandarin Stuttering Event Detection and Automatic Speech Recognition Challenge
The StutteringSpeech Challenge focuses on advancing speech technologies for people who stutter, specifically targeting Stuttering Event Detection (SED) and Automatic Speech Recognition (ASR) in Mandarin.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
NPU-NTU System for Voice Privacy 2024 Challenge
Our system employs a disentangled neural codec architecture and a serial disentanglement strategy to gradually disentangle the global speaker identity and time-variant linguistic content and paralinguistic information.
Drop the beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation
Rap, a prominent genre of vocal performance, remains underexplored in vocal generation.
Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper
Pre-trained multilingual speech foundation models, like Whisper, have shown impressive performance across different languages.
The NPU-ASLP System Description for Visual Speech Recognition in CNVSRC 2024
This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP (Team 237) in the second Chinese Continuous Visual Speech Recognition Challenge (CNVSRC 2024), engaging in all four tracks, including the fixed and open tracks of Single-Speaker VSR Task and Multi-Speaker VSR Task.
StreamVoice+: Evolving into End-to-end Streaming Zero-shot Voice Conversion
StreamVoice+ integrates a semantic encoder and a connector with the original StreamVoice framework, now trained using a non-streaming ASR.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Learning Multi-view Anomaly Detection
This study explores the recently proposed challenging multi-view Anomaly Detection (AD) task.
MUSA: Multi-lingual Speaker Anonymization via Serial Disentanglement
Meanwhile, we propose a straightforward anonymization strategy that employs empty embedding with zero values to simulate the speaker identity concealment process, eliminating the need for conversion to a pseudo-speaker identity and thereby reducing the complexity of speaker anonymization process.
AI-driven multi-omics integration for multi-scale predictive modeling of causal genotype-environment-phenotype relationships
Despite the wealth of single-cell multi-omics data, it remains challenging to predict the consequences of novel genetic and chemical perturbations in the human body.
DualVC 3: Leveraging Language Model Generated Pseudo Context for End-to-end Low Latency Streaming Voice Conversion
With speaker-independent semantic tokens to guide the training of the content encoder, the dependency on ASR is removed and the model can operate under extremely small chunks, with cascading errors eliminated.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
SCDNet: Self-supervised Learning Feature-based Speaker Change Detection
Furthermore, a contrastive learning method is proposed to mitigate the overfitting tendencies in the training of both the fine-tuning-based method and SCDNet.
Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation
The multi-codebook speech codec enables the application of large language models (LLM) in TTS but bottlenecks efficiency and robustness due to multi-sequence prediction.
AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection
The rapid advancements in speech technologies over the past two decades have led to human-level performance in tasks like automatic speech recognition (ASR) for fluent speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
BS-PLCNet 2: Two-stage Band-split Packet Loss Concealment Network with Intra-model Knowledge Distillation
Moreover, we introduce a lightweight post-processing module after packet loss restoration to recover speech distortions and remove residual noise in the audio signal.
Text-aware and Context-aware Expressive Audiobook Speech Synthesis
Recent advances in text-to-speech have significantly improved the expressiveness of synthetic speech.
WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark
Furthermore, we have created subsets of varying sizes, categorized by segment quality scores to allow for TTS model training and fine-tuning.
A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection
This paper addresses this issue by proposing a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework that is highly extensible for new methods.
Distinctive and Natural Speaker Anonymization via Singular Value Transformation-assisted Matrix
To address these issues and especially generate more natural and distinctive anonymized speech, we propose a novel speaker anonymization approach that models a matrix related to speaker identity and transforms it into an anonymized singular value transformation-assisted matrix to conceal the original speaker identity.
MMGER: Multi-modal and Multi-granularity Generative Error Correction with LLM for Joint Accent and Speech Recognition
Accents represent deviations from standard pronunciation norms, and the multi-task learning framework for simultaneous ASR and accent recognition (AR) has effectively addressed the multi-accent scenarios, making it a prominent solution.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets
Building upon this momentum, our research delves into an in-depth examination of this paradigm on a large open-source Chinese dataset.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Dynamic fault detection and diagnosis for alkaline water electrolyzer with variational Bayesian Sparse principal component analysis
Electrolytic hydrogen production serves as not only a vital source of green hydrogen but also a key strategy for addressing renewable energy consumption challenges.
Dynamic fault detection and diagnosis of industrial alkaline water electrolyzer process with variational Bayesian dictionary learning
A novel robust dynamic variational Bayesian dictionary learning (RDVDL) monitoring approach is proposed to improve the reliability and safety of AWE operation.
Nonlinear sparse variational Bayesian learning based model predictive control with application to PEMFC temperature control
Variational inference is used by NSVB-MPC to assess the predictive accuracy and make the necessary corrections to quantify system uncertainty.
MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection
Recent advancements in anomaly detection have seen the efficacy of CNN- and transformer-based approaches.
Enhancing Lip Reading with Multi-Scale Video and Multi-Encoder
Automatic lip-reading (ALR) aims to automatically transcribe spoken content from a speaker's silent lip motion captured in video.
Facilitating Reinforcement Learning for Process Control Using Transfer Learning: Perspectives
This paper provides insights into deep reinforcement learning (DRL) for process control from the perspective of transfer learning.
Anatomy-guided fiber trajectory distribution estimation for cranial nerves tractography
Diffusion MRI tractography is an important tool for identifying and analyzing the intracranial course of cranial nerves (CNs).
Approximate Message Passing-Enhanced Graph Neural Network for OTFS Data Detection
Orthogonal time frequency space (OTFS) modulation has emerged as a promising solution to support high-mobility wireless communications, for which, cost-effective data detectors are critical.
StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion
Specifically, to enable streaming capability, StreamVoice employs a fully causal context-aware LM with a temporal-independent acoustic predictor, while alternately processing semantic and acoustic features at each time step of autoregression which eliminates the dependence on complete source speech.
BS-PLCNet: Band-split Packet Loss Concealment Network with Multi-task Learning Framework and Multi-discriminators
Packet loss is a common and unavoidable problem in voice over internet phone (VoIP) systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
The NPU-ASLP-LiAuto System Description for Visual Speech Recognition in CNVSRC 2023
This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP-LiAuto (Team 237) in the first Chinese Continuous Visual Speech Recognition Challenge (CNVSRC) 2023, engaging in the fixed and open tracks of Single-Speaker VSR Task, and the open track of Multi-Speaker VSR Task.
MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition
While automatic speech recognition (ASR) systems degrade significantly in noisy environments, audio-visual speech recognition (AVSR) systems aim to complement the audio stream with noise-invariant visual cues and improve the system's robustness.
Audio-Visual Speech Recognition
Automatic Speech Recognition
+4
ICMC-ASR: The ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition Challenge
To promote speech processing and recognition research in driving scenarios, we build on the success of the Intelligent Cockpit Speech Recognition Challenge (ICSRC) held at ISCSLP 2022 and launch the ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition (ICMC-ASR) Challenge.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
LESEN: Label-Efficient deep learning for Multi-parametric MRI-based Visual Pathway Segmentation
Recent research has shown the potential of deep learning in multi-parametric MRI-based visual pathway (VP) segmentation.
Modality Exchange Network for Retinogeniculate Visual Pathway Segmentation
Secondly, we propose a cross-fusion module that further enhances the fusion of information between the two modalities.
SignGraph: A Sign Sequence is Worth Graphs of Nodes
In fact sign language tasks need to focus on the correlation of different regions in one frame and the interaction of different regions among adjacent frames for identifying a sign sequence.
U2-KWS: Unified Two-pass Open-vocabulary Keyword Spotting with Keyword Bias
Open-vocabulary keyword spotting (KWS), which allows users to customize keywords, has attracted increasingly more interest.
SELM: Speech Enhancement Using Discrete Tokens and Language Models
Given the intrinsic similarity between speech generation and speech enhancement, harnessing semantic information holds potential advantages for speech enhancement tasks.
DiAD: A Diffusion-based Framework for Multi-class Anomaly Detection
Reconstruction-based approaches have achieved remarkable outcomes in anomaly detection.
Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization
The scarcity of labeled audio-visual datasets is a constraint for training superior audio-visual speaker diarization systems.
Protein Language Model-Powered 3D Ligand Binding Site Prediction from Protein Sequence
Then we compute and update the protein-ligand interaction embedding based on the protein residue-level embeddings and ligand atom-level embeddings, and the geometric constraints in the inferred protein contact map and ligand distance map.
A Universal Framework for Accurate and Efficient Geometric Deep Learning of Molecular Systems
Molecular sciences address a wide range of problems involving molecules of different types and sizes and their complexes.
Ranked #1 on
Drug Discovery
on QM9
GPT-4V-AD: Exploring Grounding Potential of VQA-oriented GPT-4V for Zero-shot Anomaly Detection
Large Multimodal Model (LMM) GPT-4V(ision) endows GPT-4 with visual grounding capabilities, making it possible to handle certain tasks through the Visual Question Answering (VQA) paradigm.
Boosting Multi-Speaker Expressive Speech Synthesis with Semi-supervised Contrastive Learning
This paper aims to build a multi-speaker expressive TTS system, synthesizing a target speaker's speech with multiple styles and emotions.
Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation
By introducing both cross-modal and conversational representations into the decoder, our model retains context over longer sentences without information loss, achieving relative accuracy improvements of 8. 8% and 23% on Mandarin conversation datasets HKUST and MagicData-RAMC, respectively, compared to the standard Conformer model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Spike-Triggered Contextual Biasing for End-to-End Mandarin Speech Recognition
However, unlike shallow fusion methods that directly bias the posterior of the ASR model, deep biasing methods implicitly integrate contextual information, making it challenging to control the degree of bias.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
An Exploration of Task-decoupling on Two-stage Neural Post Filter for Real-time Personalized Acoustic Echo Cancellation
Utilization of speaker representation has extended the frontier of AEC, thus attracting many researchers' interest in personalized acoustic echo cancellation (PAEC).
MBTFNet: Multi-Band Temporal-Frequency Neural Network For Singing Voice Enhancement
A typical neural speech enhancement (SE) approach mainly handles speech and noise mixtures, which is not optimal for singing voice enhancement scenarios.
VITS-Based Singing Voice Conversion Leveraging Whisper and multi-scale F0 Modeling
This paper introduces the T23 team's system submitted to the Singing Voice Conversion Challenge 2023.
SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognition
Multilingual automatic speech recognition (ASR) systems have garnered attention for their potential to extend language coverage globally.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
DualVC 2: Dynamic Masked Convolution for Unified Streaming and Non-Streaming Voice Conversion
Third, the model is unable to effectively address the noise in the unvoiced segments, lowering the sound quality.
PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Prompts
In this study, we propose PromptVC, a novel style voice conversion approach that employs a latent diffusion model to generate a style vector driven by natural language prompts.
MSM-VC: High-fidelity Source Style Transfer for Non-Parallel Voice Conversion by Multi-scale Style Modeling
In addition to conveying the linguistic content from source speech to converted speech, maintaining the speaking style of source speech also plays an important role in the voice conversion (VC) task, which is essential in many scenarios with highly expressive source speech, such as dubbing and data augmentation.
Surrogate Empowered Sim2Real Transfer of Deep Reinforcement Learning for ORC Superheat Control
The Organic Rankine Cycle (ORC) is widely used in industrial waste heat recovery due to its simple structure and easy maintenance.
METTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfer
However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer - the heavy entanglement of speaker timbre, emotion, and language factors in the speech signal will make a system produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness.
Learning Universal and Robust 3D Molecular Representations with Graph Convolutional Networks
To combine the DNP descriptor and chemical features in molecules, we construct the Robust Molecular Graph Convolutional Network (RoM-GCN) which is capable to take both node and edge features into consideration when generating molecule representations.
Bundle-specific Tractogram Distribution Estimation Using Higher-order Streamline Differential Equation
At the global level, the tractography process is simplified as the estimation of bundle-specific tractogram distribution (BTD) coefficients by minimizing the energy optimization model, and is used to characterize the relations between BTD and diffusion tensor vector under the prior guidance by introducing the tractogram bundle information to provide anatomic priors.
MSW-Transformer: Multi-Scale Shifted Windows Transformer Networks for 12-Lead ECG Classification
To address these challenges, we propose a single-layer Transformer network called Multi-Scale Shifted Windows Transformer Networks (MSW-Transformer), which uses a multi-window sliding attention mechanism at different scales to capture features in different dimensions.
LM-VC: Zero-shot Voice Conversion via Speech Generation based on Language Models
An intuitive approach is to follow AudioLM - Tokenizing speech into semantic and acoustic tokens respectively by HuBERT and SoundStream, and converting source semantic tokens to target acoustic tokens conditioned on acoustic tokens of the target speaker.
Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition
By incorporating additional contextual information, deep biasing methods have emerged as a promising solution for speech recognition of personalized words.
TranUSR: Phoneme-to-word Transcoder Based Unified Speech Representation Learning for Cross-lingual Speech Recognition
Different from UniSpeech, UniData2vec replaces the quantized discrete representations with continuous and contextual representations from a teacher model for phonetically-aware pre-training.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR
The recently proposed serialized output training (SOT) simplifies multi-talker automatic speech recognition (ASR) by generating speaker transcriptions separated by a special token.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
DCCRN-KWS: an audio bias based model for noise robust small-footprint keyword spotting
Real-world complex acoustic environments especially the ones with a low signal-to-noise ratio (SNR) will bring tremendous challenges to a keyword spotting (KWS) system.
DualVC: Dual-mode Voice Conversion using Intra-model Knowledge Distillation and Hybrid Predictive Coding
Voice conversion is an increasingly popular technology, and the growing number of real-time applications requires models with streaming conversion capabilities.
Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network
In this study, we introduce a contextual phrase prediction network for an attention-based deep bias method.
Multi-level Temporal-channel Speaker Retrieval for Zero-shot Voice Conversion
Specifically, to flexibly adapt to the dynamic-variant speaker characteristic in the temporal and channel axis of the speech, we propose a novel fine-grained speaker modeling method, called temporal-channel retrieval (TCR), to find out when and where speaker information appears in speech.
Joint BS Selection, User Association, and Beamforming Design for Network Integrated Sensing and Communication
Different from conventional radar, the cellular network in the integrated sensing and communication (ISAC) system enables collaborative sensing by multiple sensing nodes, e. g., base stations (BSs).
Two-stage Neural Network for ICASSP 2023 Speech Signal Improvement Challenge
Achieving 0. 446 in the final score and 0. 517 in the P. 835 score, our system ranks 4th in the non-real-time track.
The NPU-Elevoc Personalized Speech Enhancement System for ICASSP2023 DNS Challenge
This paper describes our NPU-Elevoc personalized speech enhancement system (NAPSE) for the 5th Deep Noise Suppression Challenge at ICASSP 2023.
Two-step Band-split Neural Network Approach for Full-band Residual Echo Suppression
This paper describes a Two-step Band-split Neural Network (TBNN) approach for full-band acoustic echo cancellation.
Two Stage Contextual Word Filtering for Context bias in Unified Streaming and Non-streaming Transducer
In this paper, we propose an efficient approach to obtain a high quality contextual list for a unified streaming/non-streaming based E2E model.
A Tribute to Phil Bourne -- Scientist and Human
This Special Issue of Biomolecules, commissioned in honor of Dr. Philip E. Bourne, focuses on a new field of biomolecular data science.
MSV Challenge 2022: NPU-HC Speaker Verification System for Low-resource Indian Languages
Specifically, the weight transfer fine-tuning aims to constrain the distance of the weights between the pre-trained model and the fine-tuned model, which takes advantage of the previously acquired discriminative ability from the large-scale out-domain datasets and avoids catastrophic forgetting and overfitting at the same time.
Multi-Speaker Expressive Speech Synthesis via Multiple Factors Decoupling
This paper aims to synthesize the target speaker's speech with desired speaking style and emotion by transferring the style and emotion from reference speech recorded by other speakers.
Delivering Speaking Style in Low-resource Voice Conversion with Multi-factor Constraints
Conveying the linguistic content and maintaining the source speech's speaking style, such as intonation and emotion, is essential in voice conversion (VC).
Expressive-VC: Highly Expressive Voice Conversion with Attention Fusion of Bottleneck and Perturbation Features
We further fuse the linguistic and para-linguistic features through an attention mechanism, where speaker-dependent prosody features are adopted as the attention query, which result from a prosody encoder with target speaker embedding and normalized pitch and energy of source speech as input.
Preserving background sound in noise-robust voice conversion via multi-task learning
Background sound is an informative form of art that is helpful in providing a more immersive experience in real-application voice conversion (VC) scenarios.
Distinguishable Speaker Anonymization based on Formant and Fundamental Frequency Scaling
By directly scaling the formant and F0, the speaker distinguishability degradation of the anonymized speech caused by the introduction of other speakers is prevented.
WeKws: A production first small-footprint end-to-end Keyword Spotting Toolkit
Keyword spotting (KWS) enables speech-based user interaction and gradually becomes an indispensable component of smart devices.
Physics-aware Graph Neural Network for Accurate RNA 3D Structure Prediction
In this work, we propose a Graph Neural Network (GNN)-based scoring function trained only with the atomic types and coordinates on limited solved RNA 3D structures for distinguishing accurate structural models.
TSUP Speaker Diarization System for Conversational Short-phrase Speaker Diarization Challenge
In this challenge, we explore three kinds of typical speaker diarization systems, which are spectral clustering(SC) based diarization, target-speaker voice activity detection(TS-VAD) and end-to-end neural diarization(EEND) respectively.
spatial-dccrn: dccrn equipped with frame-level angle feature and hybrid filtering for multi-channel speech enhancement
Recently, multi-channel speech enhancement has drawn much interest due to the use of spatial information to distinguish target speech from interfering signal.
NWPU-ASLP System for the VoicePrivacy 2022 Challenge
Our system consists of four modules, including feature extractor, acoustic model, anonymization module, and neural vocoder.
ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTS
To alleviate the difficulty in training, we propose to model linguistic and prosodic information by considering cross-sentence, embedded structure in training.
Iterative Sparse Recovery based Passive Localization in Perceptive Mobile Networks
As a result, most existing methods require a large number of data samples to achieve an accurate estimate of the covariance matrix for the received signals, based on which a power spectrum is constructed for localization purposes.
The Conversational Short-phrase Speaker Diarization (CSSD) Task: Dataset, Evaluation Metric and Baselines
In the metric aspect, we design the new conversational DER (CDER) evaluation metric, which calculates the SD accuracy at the utterance level.
OLLIE: Derivation-based Tensor Program Optimizer
Boosting the runtime performance of deep neural networks (DNNs) is critical due to their wide adoption in real-world tasks.
Intelligent Reflecting Surface-Aided Maneuvering Target Sensing: True Velocity Estimation
Maneuvering target sensing will be an important service of future vehicular networks, where precise velocity estimation is one of the core tasks.
Leveraging Acoustic Contextual Representation by Audio-textual Cross-modal Learning for Conversational ASR
Then, during the training of the conversational ASR system, the extractor will be frozen to extract the textual representation of preceding speech, while such representation is used as context fed to the ASR decoder through attention mechanism.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
End-to-End Voice Conversion with Information Perturbation
The ideal goal of voice conversion is to convert the source speaker's speech to sound naturally like the target speaker while maintaining the linguistic content and the prosody of the source speech.
Efficient and Accurate Physics-aware Multiplex Graph Neural Networks for 3D Small Molecules and Macromolecule Complexes
On small molecule dataset for predicting quantum chemical properties, PaxNet reduces the prediction error by 15% and uses 73% less memory than the best baseline.
Open Source MagicData-RAMC: A Rich Annotated Mandarin Conversational(RAMC) Speech Dataset
As a Mandarin speech dataset designed for dialog scenarios with high quality and rich annotations, MagicData-RAMC enriches the data diversity in the Mandarin speech community and allows extensive research on a series of speech-related tasks, including automatic speech recognition, speaker diarization, topic detection, keyword search, text-to-speech, etc.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetings
Therefore, we propose the second approach, WD-SOT, to address alignment errors by introducing a word-level diarization model, which can get rid of such timestamp alignment dependency.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit
Recently, we made available WeNet, a production-oriented end-to-end speech recognition toolkit, which introduces a unified two-pass (U2) framework and a built-in runtime to address the streaming and non-streaming decoding modes in a single model.
An Audio-Visual Attention Based Multimodal Network for Fake Talking Face Videos Detection
DeepFake based digital facial forgery is threatening the public media security, especially when lip manipulation has been used in talking face generation, the difficulty of fake video detection is further improved.
Attention-Based Lip Audio-Visual Synthesis for Talking Face Generation in the Wild
Rather than focusing on the unimportant regions of the face image, the proposed AttnWav2Lip model is able to pay more attention on the lip region reconstruction.
Audio-visual speech separation based on joint feature representation with cross-modal attention
Multi-modal based speech separation has exhibited a specific advantage on isolating the target character in multi-talker noisy environments.
Look\&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement
Active speaker detection and speech enhancement have become two increasingly attractive topics in audio-visual scenario understanding.
Conversational Speech Recognition By Learning Conversation-level Characteristics
Conversational automatic speech recognition (ASR) is a task to recognize conversational speech including multiple speakers.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Reinforcement Learning for Personalized Drug Discovery and Design for Complex Diseases: A Systems Pharmacology Perspective
The challenges on harnessing reinforcement learning for systems pharmacology and personalized medicine are discussed.
IQDUBBING: Prosody modeling based on discrete self-supervised speech representation for expressive voice conversion
Specifically, prosody vector is first extracted from pre-trained VQ-Wav2Vec model, where rich prosody information is embedded while most speaker and environment information are removed effectively by quantization.
Multi-speaker Multi-style Text-to-speech Synthesis With Single-speaker Single-style Training Data Scenarios
Moreover, the explicit prosody features used in the prosody predicting module can increase the diversity of synthetic speech by adjusting the value of prosody features.
A Computational Efficient Maximum Likelihood Direct Position Determination Approach for Multiple Emitters Using Angle and Doppler Measurements
Emitter localization is widely applied in the military and civilian _elds.
One-shot Voice Conversion For Style Transfer Based On Speaker Adaptation
One-shot style transfer is a challenging task, since training on one utterance makes model extremely easy to over-fit to training data and causes low speaker similarity and lack of expressiveness.
Exploration of Dark Chemical Genomics Space via Portal Learning: Applied to Targeting the Undruggable Genome and COVID-19 Anti-Infective Polypharmacology
Advances in biomedicine are largely fueled by exploring uncharted territories of human biology.
Uformer: A Unet based dilated complex & real dual-path conformer network for simultaneous speech enhancement and dereverberation
Complex spectrum and magnitude are considered as two major features of speech enhancement and dereverberation.
VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis
In this paper, we propose VISinger, a complete end-to-end high-quality singing voice synthesis (SVS) system that directly generates audio waveform from lyrics and musical score.
WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition
In this paper, we present WenetSpeech, a multi-domain Mandarin corpus consisting of 10000+ hours high-quality labeled speech, 2400+ hours weakly labeled speech, and about 10000 hours unlabeled speech, with 22400+ hours in total.
Ranked #5 on
Speech Recognition
on WenetSpeech
AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person
In this paper, we present an automatic method to generate synchronized speech and talking-head videos on the basis of text and a single face image of an arbitrary person as input.
Controllable Context-aware Conversational Speech Synthesis
Specifically, we use explicit labels to represent two typical spontaneous behaviors filled-pause and prolongation in the acoustic model and develop a neural network based predictor to predict the occurrences of the two behaviors from text.
Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis
Current two-stage TTS framework typically integrates an acoustic model with a vocoder -- the acoustic model predicts a low resolution intermediate representation such as Mel-spectrum while the vocoder generates waveform from the intermediate representation.
Enriching Source Style Transfer in Recognition-Synthesis based Non-Parallel Voice Conversion
Specifically, prosodic features are used to explicit model prosody, while VAE and reference encoder are used to implicitly model prosody, which take Mel spectrum and bottleneck feature as input respectively.
Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain
Moreover, by including the data of variable numbers of speakers, our model can even better than the PIT-Conformer AR model with only 1/7 latency, obtaining WERs of 19. 9% and 34. 3% on WSJ0-2mix and WSJ0-3mix sets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
DCCRN+: Channel-wise Subband DCCRN with SNR Estimation for Speech Enhancement
Deep complex convolution recurrent network (DCCRN), which extends CRN with complex structure, has achieved superior performance in MOS evaluation in Interspeech 2020 deep noise suppression challenge (DNS2020).
Cross-Validated Tuning of Shrinkage Factors for MVDR Beamforming Based on Regularized Covariance Matrix Estimation
We propose leave-one-out cross-validation (LOOCV) choices for the shrinkage factors to optimize the beamforming performance, referred to as $\text{S}^2$CM-CV and STE-CV.
The Multi-speaker Multi-style Voice Cloning Challenge 2021
The challenge consists of two tracks, namely few-shot track and one-shot track, where the participants are required to clone multiple target voices with 100 and 5 samples respectively.
INTERSPEECH 2021 ConferencingSpeech Challenge: Towards Far-field Multi-Channel Speech Enhancement for Video Conferencing
The ConferencingSpeech 2021 challenge is proposed to stimulate research on far-field multi-channel speech enhancement for video conferencing.
Auto-KWS 2021 Challenge: Task, Datasets, and Baselines
Auto-KWS 2021 challenge calls for automated machine learning (AutoML) solutions to automate the process of applying machine learning to a customized keyword spotting task.
Regularized Covariance Estimation for Polarization Radar Detection in Compound Gaussian Sea Clutter
This paper investigates regularized estimation of Kronecker-structured covariance matrices (CM) for polarization radar in sea clutter scenarios where the data are assumed to follow the complex, elliptically symmetric (CES) distributions with a Kronecker-structured CM.
The NPU System for the 2020 Personalized Voice Trigger Challenge
This paper describes the system developed by the NPU team for the 2020 personalized voice trigger challenge.
Wake Word Detection with Streaming Transformers
Modern wake word detection systems usually rely on neural networks for acoustic modeling.
WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit
In this paper, we propose an open source, production first, and production ready speech recognition toolkit called WeNet in which a new two-pass approach is implemented to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model.
CODE-AE: A Coherent De-confounding Autoencoder for Predicting Patient-Specific Drug Response From Cell Line Transcriptomics
Thus, CODE-AE provides a useful framework to take advantage of in vitro omics data for developing generalized patient predictive models.
Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition
In this paper, we present a novel two-pass approach to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model.
Ranked #8 on
Speech Recognition
on AISHELL-1
Phonetic Posteriorgrams based Many-to-Many Singing Voice Conversion via Adversarial Training
In order to make timbre conversion more stable and controllable, speaker embedding is further decomposed to the weighted sum of a group of trainable vectors representing different timbre clusters.
TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis
On one hand, we propose to discriminate ground-truth waveform from synthetic one in frequency domain for offering more consistency guarantees instead of only in time domain.
Cascade RNN-Transducer: Syllable Based Streaming On-device Mandarin Speech Recognition with a Syllable-to-Character Converter
End-to-end models are favored in automatic speech recognition (ASR) because of its simplified system structure and superior performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Molecular Mechanics-Driven Graph Neural Network with Multiplex Graph for Molecular Structures
The prediction of physicochemical properties from molecular structures is a crucial task for artificial intelligence aided molecular design.
Ranked #2 on
Drug Discovery
on QM9
The SLT 2021 children speech recognition challenge: Open datasets, rules and baselines
Automatic speech recognition (ASR) has been significantly advanced with the use of deep learning and big data.
Sound Audio and Speech Processing
IEEE SLT 2021 Alpha-mini Speech Challenge: Open Datasets, Tracks, Rules and Baselines
In this challenge, we open source a sizable speech, keyword, echo and noise corpus for promoting data-driven methods, particularly deep-learning approaches on KWS and SSL.
Sound Audio and Speech Processing
AutoSpeech 2020: The Second Automated Machine Learning Challenge for Speech Classification
The AutoSpeech challenge calls for automated machine learning (AutoML) solutions to automate the process of applying machine learning to speech processing tasks.
A Cross-Level Information Transmission Network for Predicting Phenotype from New Genotype: Application to Cancer Precision Medicine
An unsolved fundamental problem in biology and ecology is to predict observable traits (phenotypes) from a new genetic constitution (genotype) of an organism under environmental perturbations (e. g., drug treatment).
An End-to-end Architecture of Online Multi-channel Speech Separation
Previously, we introduced a sys-tem, calledunmixing, fixed-beamformerandextraction(UFE), that was shown to be effective in addressing the speech over-lap problem in conversation transcription.
AIPerf: Automated machine learning as an AI-HPC benchmark
The de facto HPC benchmark LINPACK can not reflect AI computing power and I/O performance without representative workload.
Channel-wise Subband Input for Better Voice and Accompaniment Separation on High Resolution Music
We aim to address the major issues in CNN-based high-resolution MSS model: high computational cost and weight sharing between distinctly different bands.
Audio and Speech Processing Sound
DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement
Speech enhancement has benefited from the success of deep learning in terms of intelligibility and perceptual quality.
Ranked #8 on
Speech Enhancement
on Deep Noise Suppression (DNS) Challenge
(PESQ-NB metric)
Speech Enhancement
Audio and Speech Processing
Sound
The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Results
The paper then presents an overview of the results and system performance in the three tracks.
Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals
This model additionally has a simple and efficient stop criterion for the end of the transduction, making it able to infer the variable number of output sequences.
Ranked #3 on
Speech Separation
on WSJ0-4mix
Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition
Recently, streaming end-to-end automatic speech recognition (E2E-ASR) has gained more and more attention.
Sound Audio and Speech Processing
Simplified Self-Attention for Transformer-based End-to-End Speech Recognition
Transformer models have been introduced into end-to-end speech recognition with state-of-the-art performance on various tasks owing to their superiority in modeling long-term dependencies.
Wake Word Detection with Alignment-Free Lattice-Free MMI
Always-on spoken language interfaces, e. g. personal digital assistants, rely on a wake word to start processing spoken input.
Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
In this paper, we propose multi-band MelGAN, a much faster waveform generation model targeting to high-quality text-to-speech.
Sound Audio and Speech Processing
Adversarial Feature Learning and Unsupervised Clustering based Speech Synthesis for Found Data with Acoustic and Textual Noise
As for the speech-side noise, we propose to learn a noise-independent feature in the auto-regressive decoder through adversarial training and data augmentation, which does not need an extra speech enhancement model.
a novel cross-lingual voice cloning approach with a few text-free samples
In this paper, we present a cross-lingual voice cloning approach.
Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
We present Espresso, an open-source, modular, extensible end-to-end neural automatic speech recognition (ASR) toolkit based on the deep learning library PyTorch and the popular neural machine translation toolkit fairseq.
Ranked #1 on
Speech Recognition
on Hub5'00 CallHome
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservation
To improve the performance of attention-based GNNs, we propose cardinality preserved attention (CPA) models that can be applied to any kind of attention mechanisms.
Ranked #2 on
Graph Classification
on RE-M5K