Search Results for author:
Found 133 papers, 47 papers with code
TTS-Guided Training for Accent Conversion Without Parallel Data
Specifically, a text-to-speech (TTS) system is first pretrained with target-accented speech data.
PoE: a Panel of Experts for Generalized Automatic Dialogue Assessment
To tackle the multi-domain dialogue evaluation task, we propose a Panel of Experts (PoE), a multitask network that consists of a shared transformer encoder and a collection of lightweight adapters.
Relational Sentence Embedding for Flexible Semantic Matching
We present Relational Sentence Embedding (RSE), a new paradigm to further discover the potential of sentence embeddings.
Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation
To alleviate the adverse impact of this accumulated trajectory error, we propose a novel approach that encourages the optimization algorithm to seek a flat trajectory.
Self-Transriber: Few-shot Lyrics Transcription with Self-training
The current lyrics transcription approaches heavily rely on supervised learning with labeled data, but such data are scarce and manual labeling of singing is expensive.
I4U System Description for NIST SRE'20 CTS Challenge
This manuscript describes the I4U submission to the 2020 NIST Speaker Recognition Evaluation (SRE'20) Conversational Telephone Speech (CTS) Challenge.
ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpainting
In this paper, we study the audio-visual speaker extraction algorithms with intermittent visual cue.
Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework
Most sentence embedding techniques heavily rely on expensive human-annotated sentence pairs as the supervised signals.
token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text
Firstly, due to the distinct characteristics between speech and text modalities, where speech is continuous while text is discrete, we first discretize speech into a sequence of discrete speech tokens to solve the modality mismatch problem.
Speaker recognition with two-step multi-modal deep cleansing
However, noisy samples (i. e., with wrong labels) in the training set induce confusion and cause the network to learn the incorrect representation.
Self-Supervised Training of Speaker Encoder with Multi-Modal Diverse Positive Pairs
We study a novel neural architecture and its training strategies of speaker encoder for speaker recognition without using any identity labels.
FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis
Therefore, we propose a novel expressive conversational TTS model, termed as FCTalker, that learn the fine and coarse grained context dependency at the same time during speech generation.
Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities
Multimodal emotion recognition leverages complementary information across modalities to gain performance.
Explicit Intensity Control for Accented Text-to-speech
Accented text-to-speech (TTS) synthesis seeks to generate speech with an accent (L2) as a variant of the standard version (L1).
Mixed Emotion Modelling for Emotional Voice Conversion
Each attribute measures the degree of the relevance between the speech recordings belonging to different emotion types.
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation
Recent model-based reference-free metrics for open-domain dialogue evaluation exhibit promising correlations with human judgment.
Analyzing and Evaluating Faithfulness in Dialogue Summarization
The factual correctness of summaries has the highest priority before practical applications.
Training Spiking Neural Networks with Local Tandem Learning
The LTL rule follows the teacher-student learning approach by mimicking the intermediate feature representations of a pre-trained ANN.
A Focused Study on Sequence Length for Dialogue Summarization
Output length is critical to dialogue summarization systems.
The Kriston AI System for the VoxCeleb Speaker Recognition Challenge 2022
This technical report describes our system for track 1, 2 and 4 of the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22).
Controllable Accented Text-to-Speech Synthesis
Accented TTS synthesis is challenging as L2 is different from L1 in both in terms of phonetic rendering and prosody pattern.
The 2022 Far-field Speaker Verification Challenge: Exploring domain mismatch and semi-supervised learning under the far-field scenario
In addition, a supplementary set for the FFSVC2020 dataset is released this year.
Speech Synthesis with Mixed Emotions
We then incorporate our formulation into a sequence-to-sequence emotional text-to-speech framework.
PoLyScriber: Integrated Training of Extractor and Lyrics Transcriber for Polyphonic Music
Lyrics transcription of polyphonic music is challenging as the background music affects lyrics intelligibility.
Accurate Emotion Strength Assessment for Seen and Unseen Speech Based on Data-Driven Deep Learning
In this paper, we propose a data-driven deep learning model, i. e. StrengthNet, to improve the generalization of emotion strength assessment for seen and unseen speech.
M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database
In this work, we propose a Multi-modal Multi-scene Multi-label Emotional Dialogue dataset, M3ED, which contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9, 082 turns and 24, 449 utterances.
Music-robust Automatic Lyrics Transcription of Polyphonic Music
To improve the robustness of lyrics transcription to the background music, we propose a strategy of combining the features that emphasize the singing vocals, i. e. music-removed features that represent singing vocal extracted features, and the features that capture the singing vocals as well as the background music, i. e. music-present features.
Genre-conditioned Acoustic Models for Automatic Lyrics Transcription of Polyphonic Music
Lyrics transcription of polyphonic music is challenging not only because the singing vocals are corrupted by the background music, but also because the background music and the singing style vary across music genres, such as pop, metal, and hip hop, which affects lyrics intelligibility of the song in different ways.
A Hybrid Continuity Loss to Reduce Over-Suppression for Time-domain Target Speaker Extraction
We propose a hybrid continuity loss function for time-domain speaker extraction algorithms to settle the over-suppression problem.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data
In this way, the decoder learns to reconstruct original speech information with codes before learning to generate correct text.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Speaker Extraction with Co-Speech Gestures Cue
Speaker extraction seeks to extract the clean speech of a target speaker from a multi-talker mixture speech.
LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT
LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with the HuBERT size, achieves comparable performance to the teacher model in most tasks with a reduction of 29% parameters, and obtains a $3. 5\times$ compression ratio in three SUPERB tasks, e. g., automatic speaker verification, keyword spotting, and intent classification, with a slight accuracy loss.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Just Rank: Rethinking Evaluation with Word and Sentence Similarities
Word and sentence similarity tasks have become the de facto evaluation method.
L-SpEx: Localized Target Speaker Extraction
Speaker extraction aims to extract the target speaker's voice from a multi-talker speech mixture given an auxiliary reference utterance.
ADD 2022: the First Audio Deep Synthesis Detection Challenge
Audio deepfake detection is an emerging topic, which was included in the ASVspoof 2021.
MFA: TDNN with Multi-scale Frequency-channel Attention for Text-independent Speaker Verification with Short Utterances
The time delay neural network (TDNN) represents one of the state-of-the-art of neural solutions to text-independent speaker verification.
Emotion Intensity and its Control for Emotional Voice Conversion
As desired, the proposed network controls the fine-grained emotion intensity in the output speech.
MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation
Chatbots are designed to carry out human-like conversations across different domains, such as general chit-chat, knowledge exchange, and persona-grounded conversations.
Ranked #1 on
Dialogue Evaluation
on USR-TopicalChat
HLT-NUS SUBMISSION FOR 2020 NIST Conversational Telephone Speech SRE
This work provides a brief description of Human Language Technology (HLT) Laboratory, National University of Singapore (NUS) system submission for 2020 NIST conversational telephone speech (CTS) speaker recognition evaluation (SRE).
MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition
Multimodal emotion recognition study is hindered by the lack of labelled corpora in terms of scale and diversity, due to the high annotation cost and label ambiguity.
Disentanglement of Emotional Style and Speaker Identity for Expressive Voice Conversion
Expressive voice conversion performs identity conversion for emotional speakers by jointly converting speaker identity and emotional style.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
DeepA: A Deep Neural Analyzer For Speech And Singing Vocoding
At the same time, as the deep neural analyzer is learnable, it is expected to be more accurate for signal reconstruction and manipulation, and generalizable from speech to singing.
Self-supervised Speaker Recognition with Loss-gated Learning
In self-supervised learning for speaker recognition, pseudo labels are useful as the supervision signals.
VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over
The proposed VisualTTS adopts two novel mechanisms that are 1) textual-visual attention, and 2) visual fusion strategy during acoustic decoding, which both contribute to forming accurate alignment between the input text content and lip motion in input lip sequence.
StrengthNet: Deep Learning-based Emotion Strength Assessment for Emotional Speech Synthesis
The emotion strength of synthesized speech can be controlled flexibly using a strength descriptor, which is obtained by an emotion attribute ranking function.
Investigating the Impact of Pre-trained Language Models on Dialog Evaluation
Yet, the impact of different Pr-LMs on the performance of automatic metrics is not well-understood.
Revisiting Self-Training for Few-Shot Learning of Language Model
In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM.
PL-EESR: Perceptual Loss Based END-TO-END Robust Speaker Representation Extraction
Speech enhancement aims to improve the perceptual quality of the speech signal by suppression of the background noise.
USEV: Universal Speaker Extraction with Visual Cue
The speaker extraction algorithm requires an auxiliary reference, such as a video recording or a pre-recorded speech, to form top-down auditory attention on the target speaker.
Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification
In particular, we use synthesized speech generated from an English-Mandarin text corpus for analysis and training of a multi-lingual intent classification model.
Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification
In this regard, we leverage the reliable and widely used bidirectional encoder representations from transformers (BERT) model as a language model and transfer the knowledge to build an acoustic model for intent classification using the speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+7
Bootstrapped Unsupervised Sentence Representation Learning
As high-quality labeled data is scarce, unsupervised sentence representation learning has attracted much attention.
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers.
Serialized Multi-Layer Multi-Head Attention for Neural Speaker Embedding
Instead of utilizing multi-head attention in parallel, the proposed serialized multi-layer multi-head attention is designed to aggregate and propagate attentive statistics from one layer to the next in a serialized manner.
Expressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer
Traditional voice conversion(VC) has been focused on speaker identity conversion for speech with a neutral expression.
Selective Listening by Synchronizing Speech with Lips
A speaker extraction algorithm seeks to extract the speech of a target speaker from a multi-talker speech mixture when given a cue that represents the target speaker, such as a pre-enrolled speech utterance, or an accompanying video track.
DynaEval: Unifying Turn and Dialogue Level Evaluation
Effective evaluation metrics should reflect the dynamics of such interaction.
NUS-HLT Report for ActivityNet Challenge 2021 AVA (Speaker)
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers.
Emotional Voice Conversion: Theory, Databases and ESD
In this paper, we first provide a review of the state-of-the-art emotional voice conversion research, and the existing emotional speech databases.
The Multi-speaker Multi-style Voice Cloning Challenge 2021
The challenge consists of two tracks, namely few-shot track and one-shot track, where the participants are required to clone multiple target voices with 100 and 5 samples respectively.
Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability
To our best knowledge, this is the first study of reinforcement learning in emotional text-to-speech synthesis.
Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training
In stage 2, we perform emotion training with a limited amount of emotional speech data, to learn how to disentangle emotional style and linguistic information from the speech.
Target Speaker Verification with Selective Auditory Attention for Single and Multi-talker Speech
Inspired by the study on target speaker extraction, e. g., SpEx, we propose a unified speaker verification framework for both single- and multi-talker speech, that is able to pay selective auditory attention to the target speaker.
Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification
An intent classification system is usually implemented as a pipeline process, with a speech recognition module followed by text processing that classifies the intents.
Multi-stage Speaker Extraction with Utterance and Frame-Level Reference Signals
Speaker extraction requires a sample speech from the target speaker as the reference.
VAW-GAN for Disentanglement and Recomposition of Emotional Elements in Speech
Emotional voice conversion (EVC) aims to convert the emotion of speech from one state to another while preserving the linguistic content and speaker identity.
Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset
Emotional voice conversion aims to transform emotional prosody in speech while preserving the linguistic content and speaker identity.
GraphSpeech: Syntax-Aware Graph Attention Network For Neural Speech Synthesis
Attention-based end-to-end text-to-speech synthesis (TTS) is superior to conventional statistical methods in many ways.
Muse: Multi-modal target speaker extraction with visual cues
Speaker extraction algorithm relies on the speech sample from the target speaker as the reference point to focus its attention.
Speaker-Utterance Dual Attention for Speaker and Utterance Verification
The proposed SUDA features an attention mask mechanism to learn the interaction between the speaker and utterance information streams.
Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS
We propose a multi-task learning scheme for Tacotron training, that optimizes the system to predict both Mel spectrum and phrase breaks.
Spectrum and Prosody Conversion for Cross-lingual Voice Conversion with CycleGAN
It relies on non-parallel training data from two different languages, hence, is more challenging than mono-lingual voice conversion.
VAW-GAN for Singing Voice Conversion with Non-parallel Training Data
We train an encoder to disentangle singer identity and singing prosody (F0 contour) from phonetic content.
Multi-Tones' Phase Coding (MTPC) of Interaural Time Difference by Spiking Neural Network
Inspired by the mammal's auditory localization pathway, in this paper we propose a pure spiking neural network (SNN) based computational model for precise sound localization in the noisy real-world environment, and implement this algorithm in a real-time robotic system with a microphone array.
Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks
In this paper, we propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition, which is referred to as progressive tandem learning of deep SNNs.
Modeling Code-Switch Languages Using Bilingual Parallel Corpus
A bilingual language model is expected to model the sequential dependency for words across languages, which is difficult due to the inherent lack of suitable training data as well as diverse syntactic structure across languages.
You Only Spike Once: Improving Energy-Efficient Neuromorphic Inference to ANN-Level Accuracy
Rate-encoded SNNs could be seen as inefficient as an encoding scheme because it involves the transmission of a large number of spikes.
Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversion
We consider that there is a common code between speakers for emotional expression in a spoken language, therefore, a speaker-independent mapping between emotional states is possible.
SpEx+: A Complete Time Domain Speaker Extraction Network
To eliminate such mismatch, we propose a complete time-domain speaker extraction solution, that is called SpEx+.
Audio and Speech Processing Sound
Time-domain speaker extraction network
The inaccuracy of phase estimation is inherent to the frequency domain processing, that affects the quality of signal reconstruction.
Audio and Speech Processing Sound
SpEx: Multi-Scale Time Domain Speaker Extraction Network
Inspired by Conv-TasNet, we propose a time-domain speaker extraction network (SpEx) that converts the mixture speech into multi-scale embedding coefficients instead of decomposing the speech signal into magnitude and phase spectra.
Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks
In STDBP algorithm, the timing of individual spikes is used to convey information (temporal coding), and learning (back-propagation) is performed based on spike timing in an event-driven manner.
WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss
To address this problem, we propose a new training scheme for Tacotron-based TTS, referred to as WaveTTS, that has 2 loss functions: 1) time-domain loss, denoted as the waveform loss, that measures the distortion between the natural and generated waveform; and 2) frequency-domain loss, that measures the Mel-scale acoustic feature loss between the natural and generated acoustic features.
Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data
Many studies require parallel speech data between different emotional patterns, which is not practical in real life.
Independent language modeling architecture for end-to-end ASR
To address this problem, in this work, we propose a new architecture that separates the decoder subnet from the encoder output.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition
The brain-inspired spiking neural networks (SNN) closely mimic the biological neural networks and can operate on low-power neuromorphic hardware with spike-based computation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Teacher-Student Training for Robust Tacotron-based TTS
We first train a Tacotron2-based TTS model by always providing natural speech frames to the decoder, that serves as a teacher model.
End-to-End Code-Switching ASR for Low-Resourced Language Pairs
In this work, we describe an E2E ASR pipeline for the recognition of CS speech in which a low-resourced language is mixed with a high resourced language.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Automatic Lyrics Alignment and Transcription in Polyphonic Music: Does Background Music Help?
Automatic lyrics alignment and transcription in polyphonic music are challenging tasks because the singing vocals are corrupted by the background music.
Audio and Speech Processing Sound
Neural Population Coding for Effective Temporal Classification
We show that, with population neural codings, the encoded patterns are linearly separable using the Support Vector Machine (SVM).
An efficient and perceptually motivated auditory neural encoding and decoding algorithm for spiking neural networks
The neural encoding scheme, that we call Biologically plausible Auditory Encoding (BAE), emulates the functions of the perceptual components of the human auditory system, that include the cochlear filter bank, the inner hair cells, auditory masking effects from psychoacoustic models, and the spike neural encoding by the auditory nerve.
A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks
Spiking neural networks (SNNs) represent the most prominent biologically inspired computing model for neuromorphic computing (NC) architectures.
Acoustic Modeling for Automatic Lyrics-to-Audio Alignment
In this work, we propose (1) using additional speech and music-informed features and (2) adapting the acoustic models trained on a large amount of solo singing vocals towards polyphonic music using a small amount of in-domain data.
Large-Scale Speaker Diarization of Radio Broadcast Archives
This paper describes our initial efforts to build a large-scale speaker diarization (SD) and identification system on a recently digitized radio broadcast archive from the Netherlands which has more than 6500 audio tapes with 3000 hours of Frisian-Dutch speech recorded between 1950-2016.
Code-Switching Detection Using ASR-Generated Language Posteriors
Code-switching (CS) detection refers to the automatic detection of language switches in code-mixed utterances.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Multi-Graph Decoding for Code-Switching ASR
This archive contains recordings with monolingual Frisian and Dutch speech segments as well as Frisian-Dutch CS speech, hence the recognition performance on monolingual segments is also vital for accurate transcriptions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019
Our proposed approach significantly improved the intelligibility (in CER), the MOS, and discrimination ABX scores compared to the official ZeroSpeech 2019 baseline or even the topline.
I4U Submission to NIST SRE 2018: Leveraging from a Decade of Shared Experiences
The I4U consortium was established to facilitate a joint entry to NIST speaker recognition evaluations (SRE).
Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet
We propose using an extended model architecture of Tacotron, that is a multi-source sequence-to-sequence model with a dual attention mechanism as the shared model for both the TTS and VC tasks.
Optimization of Speaker Extraction Neural Network with Magnitude and Temporal Spectrum Approximation Loss
The SpeakerBeam-FE (SBF) method is proposed for speaker extraction.
Deep Spiking Neural Network with Spike Count based Learning Rule
Deep spiking neural networks (SNNs) support asynchronous event-driven computation, massive parallelism and demonstrate great potential to improve the energy efficiency of its synchronous analog counterpart.
On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition
Code-switching (CS) refers to a linguistic phenomenon where a speaker uses different languages in an utterance or between alternating utterances.
Generative x-vectors for text-independent speaker verification
Speaker verification (SV) systems using deep neural network embeddings, so-called the x-vector systems, are becoming popular due to its good performance superior to the i-vector systems.
Is Neuromorphic MNIST neuromorphic? Analyzing the discriminative power of neuromorphic datasets in the time domain
We also use this SNN for further experiments on N-MNIST to show that rate based SNNs perform better, and precise spike timings are not important in N-MNIST.
Named-Entity Tagging and Domain adaptation for Better Customized Translation
Customized translation need pay spe-cial attention to the target domain ter-minology especially the named-entities for the domain.
Report of NEWS 2018 Named Entity Transliteration Shared Task
This report presents the results from the Named Entity Transliteration Shared Task conducted as part of The Seventh Named Entities Workshop (NEWS 2018) held at ACL 2018 in Melbourne, Australia.
Learning Acoustic Word Embeddings with Temporal Context for Query-by-Example Speech Search
We also find that it is important to have sufficient speech segment pairs to train the deep CNN for effective acoustic word embeddings.
A Multi-State Diagnosis and Prognosis Framework with Feature Learning for Tool Condition Monitoring
For fault diagnosis, a cost-sensitive deep belief network (namely ECS-DBN) is applied to deal with the imbalanced data problem for tool state estimation.
A Cost-Sensitive Deep Belief Network for Imbalanced Classification
Adaptive differential evolution optimization is implemented as the optimization algorithm that automatically updates its corresponding parameters without the need of prior domain knowledge.
Statistical Parametric Speech Synthesis Using Generative Adversarial Networks Under A Multi-task Learning Framework
In this paper, we aim at improving the performance of synthesized speech in statistical parametric speech synthesis (SPSS) based on a generative adversarial network (GAN).
Sound
Spoofing detection under noisy conditions: a preliminary investigation and an initial database
To simulate the real-life scenarios, we perform a preliminary investigation of spoofing detection under additive noisy conditions, and also describe an initial database for this task.
Fantastic 4 system for NIST 2015 Language Recognition Evaluation
This article describes the systems jointly submitted by Institute for Infocomm (I$^2$R), the Laboratoire d'Informatique de l'Universit\'e du Maine (LIUM), Nanyang Technology University (NTU) and the University of Eastern Finland (UEF) for 2015 NIST Language Recognition Evaluation (LRE).
The NNI Query-by-Example System for MediaEval 2015
This paper describes the system developed by the NNI team for the Query-by-Example Search on Speech Task (QUESST) in the MediaEval 2015 evaluation.
Ranked #9 on
Keyword Spotting
on QUESST
The NNI Query-by-Example System for MediaEval 2014
For both symbolic and DTW search, partial sequence matching is performed to reduce missing rate, especially for query type 2 and 3.
Ranked #6 on
Keyword Spotting
on QUESST
