Search Results for author:
Found 229 papers, 91 papers with code
Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models’ Transferability
This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications.
Self-supervised Representation Learning for Speech Processing
Due to the growing popularity of SSL, and the shared mission of the areas in bringing speech and language technologies to more use cases with better quality and scaling the technologies for under-represented languages, we propose this tutorial to systematically survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing.
A Large-Scale Evaluation of Speech Foundation Models
In this work, we establish the Speech processing Universal PERformance Benchmark (SUPERB) to study the effectiveness of the paradigm for speech.
Codec-SUPERB: An In-Depth Analysis of Sound Codec Models
The sound codec's dual roles in minimizing data transmission latency and serving as tokenizers underscore its critical importance.
Towards audio language modeling - an overview
Neural audio codecs are initially introduced to compress audio data into compact codes to reduce transmission latency.
Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations
When using text-only LLMs to model spoken dialogue, text-only LLMs cannot give different responses based on the speaking style of the current turn.
EMO-SUPERB: An In-depth Look at Speech Emotion Recognition
Speech emotion recognition (SER) is a pivotal technology for human-computer interaction systems.
SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data
Second, we propose a new hybrid architecture that merges the cascaded and parallel architectures of SpeechCLIP into a multi-task learning framework.
Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations
We show that LLMs can generate paragraphs that contain verifiable facts, but the facts are combined to form a non-factual paragraph due to entity ambiguity.
Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model
Recent advances in self-supervised speech models have shown significant improvement in many downstream tasks.
REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR
A word/phoneme in the speech signal is represented by a segment of speech signal with variable length and unknown boundary, and this segmental structure makes learning the mapping between speech and text challenging, especially without paired data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
SpeechDPR: End-to-End Spoken Passage Retrieval for Open-Domain Spoken Question Answering
However, the real-world problem of Open-domain SQA (openSQA), in which the machine needs to first retrieve passages that possibly contain the answer from a spoken archive in addition, was never considered.
Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initialization
This paper presents an effective transfer learning framework for language adaptation in text-to-speech systems, with a focus on achieving language adaptation using minimal labeled and unlabeled data.
Over-Reasoning and Redundant Calculation of Large Language Models
Large language models (LLMs) can solve problems step-by-step.
Examining Forgetting in Continual Pre-training of Aligned Large Language Models
Recent advances in Large Language Models (LLMs) have exhibited remarkable proficiency across various tasks.
Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks
In the realm of spoken language understanding (SLU), numerous natural language understanding (NLU) methodologies have been adapted by supplying large language models (LLMs) with transcribed speech instead of conventional written text.
PEFT for Speech: Unveiling Optimal Placement, Merging Strategies, and Ensemble Techniques
Our study conducts extensive experiments to compare different PEFT methods and their layer-wise placement adapting Differentiable Architecture Search (DARTS).
Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision
This work evaluated several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora.
Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue
Specifically, our framework serializes tasks in the order of current paralinguistic attribute prediction, response paralinguistic attribute prediction, and response text generation with autoregressive conditioning.
GSQA: An End-to-End Model for Generative Spoken Question Answering
Furthermore, the GSQA model has only been fine-tuned on the spoken extractive QA dataset.
Scalable Ensemble-based Detection Method against Adversarial Attacks for speaker verification
Automatic speaker verification (ASV) is highly susceptible to adversarial attacks.
An Exploration of In-Context Learning for Speech Language Model
Despite the success of ICL in NLP, little work is exploring the possibility of ICL in speech processing.
Learning from Red Teaming: Gender Bias Provocation and Mitigation in Large Language Models
Recently, researchers have made considerable improvements in dialogue systems with the progress of large language models (LLMs) such as ChatGPT and GPT-4.
Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond
The 2023 Multilingual Speech Universal Performance Benchmark (ML-SUPERB) Challenge expands upon the acclaimed SUPERB framework, emphasizing self-supervised models in multilingual speech recognition and language identification.
A Closer Look into Automatic Evaluation Using Large Language Models
In this paper, we analyze LLM evaluation (Chiang and Lee, 2023) and G-Eval (Liu et al., 2023), and we discuss how those details in the evaluation process change how well the ratings given by LLMs correlate with human ratings.
Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages
By simply adding the chat vector to a continual pre-trained model's weights, we can endow the model with chat capabilities in new languages without the need for further training.
Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Model
Notably, in the low-resource scenario, prompting consistently outperforms adapter tuning.
Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languages
We introduce a new zero resource code-switched speech benchmark designed to directly assess the code-switching capabilities of self-supervised speech encoders.
Low-Resource Self-Supervised Learning with SSL-Enhanced TTS
Self-supervised learning (SSL) techniques have achieved remarkable results in various speech processing tasks.
Investigating Human-Identifiable Features Hidden in Adversarial Perturbations
Our study contributes to a deeper understanding of the underlying mechanisms behind adversarial attacks and offers insights for the development of more resilient defense strategies for neural networks.
Towards General-Purpose Text-Instruction-Guided Voice Conversion
This paper introduces a novel voice conversion (VC) model, guided by text instructions such as "articulate slowly with a deep tone" or "speak in a cheerful boyish voice".
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information.
Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
To achieve comprehensive coverage of diverse speech tasks and harness instruction tuning, we invite the community to collaborate and contribute, facilitating the dynamic growth of the benchmark.
Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC
In our experiments, our model outperforms the baseline autoregressive model (Transformer \textit{base}) on multiple datasets, including WMT'14 DE$\leftrightarrow$EN, WMT'16 RO$\leftrightarrow$EN, and IWSLT'14 DE$\leftrightarrow$EN.
Language Modelling
Pretrained Multilingual Language Models
+1
Why We Should Report the Details in Subjective Evaluation of TTS More Rigorously
This paper emphasizes the importance of reporting experiment details in subjective evaluations and demonstrates how such details can significantly impact evaluation results in the field of speech synthesis.
SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts
In this paper, we present pioneering research that explores the application of prompt tuning to stimulate speech LMs for various generation tasks, within a unified framework called SpeechGen, with around 10M trainable parameters.
How to Estimate Model Transferability of Pre-Trained Speech Models?
In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks.
MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Models
In this paper, we introduce MiniSUPERB, a lightweight benchmark that efficiently evaluates SSL speech models with comparable results to SUPERB but lower computational costs significantly.
Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target
However, the absence of intermediate targets and training guidance for textless SLU often results in suboptimal performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
The defender's perspective on automatic speaker verification: An overview
Automatic speaker verification (ASV) plays a critical role in security-sensitive environments.
ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks.
Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation
We use fully unpaired data to train our unsupervised systems and evaluate our results on CoVoST 2 and CVSS.
Can Large Language Models Be an Alternative to Human Evaluations?
We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.
Cascading and Direct Approaches to Unsupervised Constituency Parsing on Spoken Sentences
The goal is to determine the spoken sentences' hierarchical syntactic structure in the form of constituency parse trees, such that each node is a span of audio that corresponds to a constituent.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks
For speech processing, SpeechPrompt shows its high parameter efficiency and competitive performance on a few speech classification tasks.
Ranked #17 on
Spoken Language Understanding
on Fluent Speech Commands
(using extra training data)
Ensemble knowledge distillation of self-supervised speech models
We tried two different aggregation techniques, layerwise-average and layerwise-concatenation, to the representations of different teacher models and found that the former was more effective.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue
With the power of large pretrained language models, various research works have integrated knowledge into dialogue systems.
Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs
Aiming to produce reinforcement learning (RL) policies that are human-interpretable and can generalize better to novel scenarios, Trivedi et al. (2021) present a method (LEAPS) that first learns a program embedding space to continuously parameterize diverse programs from a pre-generated program dataset, and then searches for a task-solving program in the learned program embedding space when given a task.
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks
In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape.
General Framework for Self-Supervised Model Priming for Parameter-Efficient Fine-tuning
To tackle these issues, we propose a general framework to enhance the few-shot adaptation and cross-domain generalization ability of parameter-efficient methods.
CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Models
However, such efficient tuning techniques only provide adaptation at the transformer layer, but failed to perform adaptation at the feature extractor.
EURO: ESPnet Unsupervised ASR Open-source Toolkit
This paper describes the ESPnet Unsupervised ASR Open-source Toolkit (EURO), an end-to-end open-source toolkit for unsupervised automatic speech recognition (UASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Model Extraction Attack against Self-supervised Speech Models
In the first stage, SSL is conducted on the large-scale unlabeled corpus to pre-train a small speech model.
MelHuBERT: A simplified HuBERT on Mel spectrograms
Self-supervised models have had great success in learning speech representations that can generalize to various downstream tasks.
Compressing Transformer-based self-supervised models for speech processing
Despite the success of Transformers in self- supervised learning with applications to various downstream tasks, the computational cost of training and inference remains a major challenge for applying these models to a wide spectrum of devices.
Introducing Semantics into Speech Encoders
Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+10
Bridging Speech and Textual Pre-trained Models with Unsupervised ASR
To be specific, we propose to use unsupervised automatic speech recognition (ASR) as a connector that bridges different modalities used in speech and textual pre-trained models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Once-for-All Sequence Compression for Self-Supervised Speech Models
The sequence length along the time axis is often the dominant factor of the computation in speech processing.
M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval
This work investigates the use of large-scale, English-only pre-trained models (CLIP and HuBERT) for multilingual image-speech retrieval.
T5lephone: Bridging Speech and Text Self-supervised Models for Spoken Language Understanding via Phoneme level T5
In Spoken language understanding (SLU), a natural solution is concatenating pre-trained speech models (e. g. HuBERT) and pretrained language models (PLM, e. g. T5).
Multimodal Transformer Distillation for Audio-Visual Synchronization
This paper proposed an MTDVocaLiST model, which is trained by our proposed multimodal Transformer distillation (MTD) loss.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We present the SUPERB challenge at SLT 2022, which aims at learning self-supervised speech representation for better performance, generalization, and efficiency.
Improving generalizability of distilled self-supervised speech processing models under distorted settings
Self-supervised learned (SSL) speech pre-trained models perform well across various speech processing tasks.
On the Utility of Self-supervised Models for Prosody-related Tasks
We find that 13 of the 15 SSL models outperformed the baseline on all the prosody-related tasks.
On Compressing Sequences for Self-Supervised Speech Models
Subsampling while training self-supervised models not only improves the overall performance on downstream tasks under certain frame rates, but also brings significant speed-up in inference.
Exploring Efficient-tuning Methods in Self-supervised Speech Models
In downstream tasks, the parameters of SSL models are frozen, and only the adapters are trained.
Are Synonym Substitution Attacks Really Synonym Substitution Attacks?
In this paper, we explore the following question: Are synonym substitution attacks really synonym substitution attacks (SSAs)?
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly.
Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection
Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications.
Adversarial Robustness
Audio-Visual Active Speaker Detection
The Efficacy of Self-Supervised Speech Models for Audio Representations
Extensive experiments on speech and non-speech audio datasets are conducted to investigate the representation abilities of our ensemble method and its single constituent model.
Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network
GAN training is adopted in the first stage to find the mapping relationship between unpaired speech and phone sequence.
Few-Shot Cross-Lingual TTS Using Transferable Phoneme Embedding
This paper studies a transferable phoneme embedding framework that aims to deal with the cross-lingual text-to-speech (TTS) problem under the few-shot setting.
Tackling Spoofing-Aware Speaker Verification with Multi-Model Fusion
However, previous works show that state-of-the-art ASV models are seriously vulnerable to voice spoofing attacks, and the recently proposed high-performance spoofing countermeasure (CM) models only focus solely on the standalone anti-spoofing tasks, and ignore the subsequent speaker verification process.
Learning to Generate Prompts for Dialogue Generation through Reinforcement Learning
Much literature has shown that prompt-based learning is an efficient method to make use of the large pre-trained language model.
Searching for the Essence of Adversarial Perturbations
Neural networks have demonstrated state-of-the-art performance in various machine learning fields.
Structured Prompt Tuning
We propose structured prompt tuning, a simple and effective method to improve prompt tuning.
Self-Supervised Speech Representation Learning: A Review
Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Silence is Sweeter Than Speech: Self-Supervised Model Using Silence to Store Speaker Information
We found that HuBERT stores speaker information in representations whose positions correspond to silences in a waveform.
Meta Learning for Natural Language Processing: A Survey
Deep learning has been the mainstream technique in natural language processing (NLP) area.
AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks
Transformer-based pre-trained models with millions of parameters require large storage.
XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding
Transformer-based models are widely used in natural language understanding (NLU) tasks, and multimodal transformers have been effective in visual-language tasks.
Re-Examining Human Annotations for Interpretable NLP
Our results reveal that the annotation quality is highly subject to the workers' qualification, and workers can be guided to provide certain annotations by the instructions.
Understanding, Detecting, and Separating Out-of-Distribution Samples and Adversarial Samples in Text Classification
Based on our observations, we propose a simple method to separate ID, OOD, and Adv samples using the hidden representations and output probabilities of the model.
DDOS: A MOS Prediction Framework utilizing Domain Adaptive Pre-training and Distribution of Opinion Scores
Mean opinion score (MOS) is a typical subjective evaluation metric for speech synthesis systems.
On the Efficiency of Integrating Self-supervised Learning and Meta-learning for User-defined Few-shot Keyword Spotting
User-defined keyword spotting is a task to detect new spoken terms defined by users.
Universal Adaptor: Converting Mel-Spectrograms Between Different Configurations for Speech Synthesis
Most recent speech synthesis systems are composed of a synthesizer and a vocoder.
SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks
We report in this paper the first exploration of the prompt tuning paradigm for speech processing tasks based on Generative Spoken Language Model (GSLM).
Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation
Speech distortions are a long-standing problem that degrades the performance of supervisely trained speech processing models.
Spoofing-Aware Speaker Verification by Multi-Level Fusion
In the second-level fusion, the CM score and ASV scores directly from ASV systems will be concatenated into a prediction block for the final decision.
Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognition
Although deep learning-based end-to-end Automatic Speech Recognition (ASR) has shown remarkable performance in recent years, it suffers severe performance regression on test samples drawn from different data distributions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation
Simultaneous speech translation (SimulST) is a challenging task aiming to translate streaming speech before the complete input is observed.
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities
In this paper, we introduce SUPERB-SG, a new benchmark focused on evaluating the semantic and generative capabilities of pre-trained models by increasing task diversity and difficulty over SUPERB.
DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering
We empirically showed that DUAL yields results comparable to those obtained by cascading ASR and text QA model and robust to real-world data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Partially Fake Audio Detection by Self-attention-based Fake Span Discovery
Also ADD 2022 is the first challenge to propose the partially fake audio detection task.
Anticipation-Free Training for Simultaneous Machine Translation
Existing methods increase latency or introduce adaptive read-write policies for SimulMT models to handle local reordering and improve translation quality.
Membership Inference Attacks Against Self-supervised Speech Models
Recently, adapting the idea of self-supervised learning (SSL) on continuous speech has started gaining attention.
Characterizing the adversarial vulnerability of speech self-supervised learning
As the paradigm of the self-supervised learning upstream model followed by downstream tasks arouses more attention in the speech community, characterizing the adversarial robustness of such paradigm is of high priority.
Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
On the one hand, speaker adaptation methods fine-tune a trained multi-speaker text-to-speech (TTS) model with few enrolled samples.
Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuning
It has been shown that an SSL pretraining model can achieve excellent performance in various downstream tasks of speech processing.
Don't speak too fast: The impact of data bias on self-supervised speech models
Self-supervised Speech Models (S3Ms) have been proven successful in many speech downstream tasks, like ASR.
Toward Degradation-Robust Voice Conversion
However, in real-world scenarios, it is difficult to collect clean utterances of a speaker, and they are usually degraded by noises or reverberations.
S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations
In this work, we provide a series of in-depth analyses by benchmarking on the two tasks in VCC2020, namely intra-/cross-lingual any-to-one (A2O) VC, as well as an any-to-any (A2A) setting.
An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning
Apart from the coherence and fluency of responses, an empathetic chatbot emphasizes more on people's feelings.
Analyzing the Robustness of Unsupervised Speech Recognition
In this work, we further analyze the training robustness of unsupervised ASR on the domain mismatch scenarios in which the domains of unpaired speech and text are different.
Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models
Code-switching (CS) is common in daily conversations where more than one language is used within a sentence.
DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT
Self-supervised speech representation learning methods like wav2vec 2. 0 and Hidden-unit BERT (HuBERT) leverage unlabeled speech data for pre-training and offer good representations for numerous speech processing tasks.
On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets
In this work, we study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
Parallelized Reverse Curriculum Generation
For reinforcement learning (RL), it is challenging for an agent to master a task that requires a specific series of actions due to sparse rewards.
Meta Learning and Its Applications to Natural Language Processing
Meta-learning is one of the most important new techniques in machine learning in recent years.
Adversarial Sample Detection for Speaker Verification by Neural Vocoders
We also show that the neural vocoder adopted in the detection framework is dataset-independent.
Voting for the right answer: Adversarial defense for speaker verification
Automatic speaker verification (ASV) is a well developed technology for biometric identification, and has been ubiquitous implemented in security-critic applications, such as banking and access control.
Mitigating Biases in Toxic Language Detection through Invariant Rationalization
Automatic detection of toxic language plays an essential role in protecting social media users, especially minority groups, from verbal abuse.
Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning
This work is among the first to perform adversarial defense for ASV without knowing the specific attack algorithms.
Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation
We study the possibilities of building a non-autoregressive speech-to-text translation model using connectionist temporal classification (CTC), and use CTC-based automatic speech recognition as an auxiliary task to improve the performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Improving Cross-Lingual Reading Comprehension with Self-Training
Substantial improvements have been made in machine reading comprehension, where the machine answers questions based on a given context.
SpeechNet: A Universal Modularized Model for Speech Processing Tasks
The multi-task learning of a wide variety of speech processing tasks with a universal model has not been studied.
SUPERB: Speech processing Universal PERformance Benchmark
SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data.
S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations
AUTOVC used dvector to extract speaker information, and self-supervised learning (SSL) features like wav2vec 2. 0 is used in FragmentVC to extract the phonetic content information.
Utilizing Self-supervised Representations for MOS Prediction
In this paper, we use self-supervised pre-trained models for MOS prediction.
Non-autoregressive Mandarin-English Code-switching Speech Recognition
Mandarin-English code-switching (CS) is frequently used among East and Southeast Asian people.
Towards Lifelong Learning of End-to-end ASR
We can collect new data describing the new environment and fine-tune the system, but this naturally leads to higher error rates for the earlier datasets, referred to as catastrophic forgetting.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Auto-KWS 2021 Challenge: Task, Datasets, and Baselines
Auto-KWS 2021 challenge calls for automated machine learning (AutoML) solutions to automate the process of applying machine learning to a customized keyword spotting task.
Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention
Our framework includes a guiding chatbot and an interlocutor model that plays the role of humans.
Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability
This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications.
Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech
The few-shot multi-speaker multi-style voice cloning task is to synthesize utterances with voice and speaking style similar to a reference speaker given only a few reference samples.
Adversarial defense for automatic speaker verification by cascaded self-supervised learning models
Automatic speaker verification (ASV) is one of the core technologies in biometric identification.
Pre-Training a Language Model Without Human Language
In this paper, we study how the intrinsic nature of pre-training data contributes to the fine-tuned downstream performance.
TaylorGAN: Neighbor-Augmented Policy Update Towards Sample-Efficient Natural Language Generation
Score function-based natural language generation (NLG) approaches such as REINFORCE, in general, suffer from low sample efficiency and training instability problems.
TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation
Score function-based natural language generation (NLG) approaches such as REINFORCE, in general, suffer from low sample efficiency and training instability problems.
How Far Are We from Robust Voice Conversion: A Survey
Voice conversion technologies have been greatly improved in recent years with the help of deep learning, but their capabilities of producing natural sounding utterances in different conditions remain unclear.
Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis
Prosody modeling is an essential component in modern text-to-speech (TTS) frameworks.
AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization
With a proper activation as an information bottleneck on content embeddings, the trade-off between the synthesis quality and the speaker similarity of the converted speech is improved drastically.
Audio and Speech Processing Sound
Stabilizing Label Assignment for Speech Separation by Self-supervised Pre-training
Speech separation has been well developed, with the very successful permutation invariant training (PIT) approach, although the frequent label assignment switching happening during PIT training remains to be a problem when better convergence speed and achievable performance are desired.
Ranked #6 on
Speech Separation
on Libri2Mix
(using extra training data)
FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention
Any-to-any voice conversion aims to convert the voice from and to any speakers even unseen during training, which is much more challenging compared to one-to-one or many-to-many tasks, but much more attractive in real-world scenarios.
Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data.
Unsupervised Multiple Choices Question Answering: Start Learning from Basic Knowledge
In this paper, we study the possibility of almost unsupervised Multiple Choices Question Answering (MCQA).
What makes multilingual BERT multilingual?
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings.
Looking for Clues of Language in Multilingual BERT to Improve Cross-lingual Generalization
Token embeddings in multilingual BERT (m-BERT) contain both language and semantic information.
Pretrained Language Model Embryology: The Birth of ALBERT
These findings suggest that knowledge of a pretrained model varies during pretraining, and having more pretrain steps does not necessarily provide a model with more comprehensive knowledge.
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
We present a large-scale comparison of various self-supervised models.
Investigation of Sentiment Controllable Chatbot
Conventional seq2seq chatbot models attempt only to find sentences with the highest probabilities conditioned on the input sequences, without considering the sentiment of the output sentences.
Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers
In this paper, we seek solutions for reducing the computation complexity of transformer-based models for speech representation learning.
VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture
Voice conversion (VC) is a task that transforms the source speaker's timbre, accent, and tones in audio into another one's while preserving the linguistic content.
Understanding Self-Attention of Self-Supervised Audio Transformers
Self-supervised Audio Transformers (SAT) enable great success in many downstream speech applications like ASR, but how they work has not been widely explored yet.
Defense for Black-box Attacks on Anti-spoofing Models by Self-Supervised Learning
To explore this issue, we proposed to employ Mockingjay, a self-supervised learning based model, to protect anti-spoofing models against adversarial attacks in the black-box scenario.
Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation
Speech translation (ST) aims to learn transformations from speech in the source language to the text in the target language.
Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
We use the representations with two downstream tasks, speaker identification, and phoneme classification.
Defending Your Voice: Adversarial Attack on Voice Conversion
We introduce human imperceptible noise into the utterances of a speaker whose voice is to be defended.
Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation
The experiment results demonstrate that with only an hour of paired speech data, no matter the paired data is from multiple speakers or a single speaker, the proposed model can generate intelligible speech in different voices.
WG-WaveNet: Real-Time High-Fidelity Speech Synthesis without GPU
As we design a flow-based model that is heavily compressed, the proposed model requires much less computational resources compared to other waveform generation models during both training and inference time; even though the model is highly compressed, the post-filter maintains the quality of generated waveform.
Speech Synthesis
Text-To-Speech Synthesis
Audio and Speech Processing
Sound
DARTS-ASR: Differentiable Architecture Search for Multilingual Speech Recognition and Adaptation
In order to examine the generalizability of DARTS-ASR, we apply our approach not only on many languages to perform monolingual ASR, but also on a multilingual ASR setting.
End-to-end Whispered Speech Recognition with Frequency-weighted Approaches and Pseudo Whisper Pre-training
Whispering is an important mode of human speech, but no end-to-end recognition results for it were reported yet, probably due to the scarcity of available whispered speech data.
Learning Interpretable and Discrete Representations with Adversarial Training for Unsupervised Text Classification
Also, the performance is on par with a recently proposed weakly-supervised text classification method.
A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings.
Defense against adversarial attacks on spoofing countermeasures of ASV
Various forefront countermeasure methods for automatic speaker verification (ASV) with considerable performance in anti-spoofing are proposed in the ASVspoof 2019 challenge.
BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
Although Bidirectional Encoder Representations from Transformers (BERT) have achieved tremendous success in many natural language processing (NLP) tasks, it remains a black box.
MITAS: A Compressed Time-Domain Audio Separation Network with Parameter Sharing
Our experimental results first confirmed the robustness of our MiTAS on two types of perturbations in mixed audio.
Towards Robust Neural Vocoding for Speech Generation: A Survey
We found out that the speaker variety is much more important for achieving a universal vocoder than the language.
J-Net: Randomly weighted U-Net for audio source separation
According to these discoveries, we pose two questions: what is the value of randomly weighted networks in difficult generative audio tasks such as audio source separation and does such positive correlation still exist when it comes to large random networks and their trained counterparts?
Training a code-switching language model with monolingual data
A lack of code-switching data complicates the training of code-switching (CS) language models.
What does a network layer hear? Analyzing hidden representations of end-to-end ASR through speech synthesis
End-to-end speech recognition systems have achieved competitive results compared to traditional systems.
Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning
In this paper we propose a Sequential Representation Quantization AutoEncoder (SeqRQ-AE) to learn from primarily unpaired audio data and produce sequences of representations very close to phoneme sequences of speech utterances.
Sequence-to-sequence Automatic Speech Recognition with Word Embedding Regularization and Fused Decoding
This allows the decoder to consider the semantic consistency during decoding by absorbing the information carried by the transformed decoder feature, which is learned to be close to the target word embedding.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Interrupted and cascaded permutation invariant training for speech separation
Permutation Invariant Training (PIT) has long been a stepping stone method for training speech separation model in handling the label ambiguity problem.
Ranked #22 on
Speech Separation
on WSJ0-2mix
Meta Learning for End-to-End Low-Resource Speech Recognition
In this paper, we proposed to apply meta learning approach for low-resource automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
In addition to the potential of end-to-end SQA, the SpeechBERT can also be considered for many other spoken language understanding tasks just as BERT for many text processing tasks.
Ranked #3 on
Spoken Language Understanding
on Spoken-SQuAD
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech.
Adversarial Attacks on Spoofing Countermeasures of automatic speaker verification
High-performance spoofing countermeasure systems for automatic speaker verification (ASV) have been proposed in the ASVspoof 2019 challenge.
DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphs
This paper proposes a new task about how to apply dynamic knowledge graphs in neural conversation model and presents a novel TV series conversation corpus (DyKgChat) for the task.
Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
Because it is not feasible to collect training data for every language, there is a growing interest in cross-lingual transfer learning.
Tree Transformer: Integrating Tree Structures into Self-Attention
This paper proposes Tree Transformer, which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures.
Order-free Learning Alleviating Exposure Bias in Multi-label Classification
In this paper, we propose a new framework for MLC which does not rely on a predefined label order and thus alleviates exposure bias.
LAMOL: LAnguage MOdeling for Lifelong Language Learning
We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling.
Ranked #4 on
Continual Learning
on ASC (19 tasks)
Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets
Paraphrase generation is an interesting and challenging NLP task which has numerous practical applications.
Cross-Lingual Transfer Learning for Question Answering
In this paper, we explore the problem of cross-lingual transfer learning for QA, where a source language task with plentiful annotations is utilized to improve the performance of a QA model on a target language task with limited available annotations.
Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion
We found that the proposed encoding method offers automatic extraction of speech content from speaker style, and is sufficient to cover full linguistic content in a given language.
Mitigating the Impact of Speech Recognition Errors on Spoken Question Answering by Adversarial Domain Adaptation
Spoken question answering (SQA) is challenging due to complex reasoning on top of the spoken documents.
Ranked #3 on
Spoken Language Understanding
on Spoken-SQuAD
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Improved Speech Separation with Time-and-Frequency Cross-domain Joint Embedding and Clustering
Substantial effort has been reported based on approaches over spectrogram, which is well known as the standard time-and-frequency cross-domain representation for speech signals.
Ranked #24 on
Speech Separation
on WSJ0-2mix
End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning
In this paper, we aim to build TTS systems for such low-resource (target) languages where only very limited paired data are available.
One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
Recently, voice conversion (VC) without parallel data has been successfully adapted to multi-target scenario in which a single model is trained to convert the input voice to many different speakers.
From Semi-supervised to Almost-unsupervised Speech Recognition with Very-low Resource by Jointly Learning Phonetic Structures from Audio and Text Embeddings
However, we note human babies start to learn the language by the sounds (or phonetic structures) of a small number of exemplar words, and "generalize" such knowledge to other words without hearing a large amount of data.
Completely Unsupervised Speech Recognition By A Generative Adversarial Network Harmonized With Iteratively Refined Hidden Markov Models
Producing a large annotated speech corpus for training ASR systems remains difficult for more than 95% of languages all over the world which are low-resourced, but collecting a relatively big unlabeled data set for such languages is more achievable.
Adversarial Learning of Label Dependency: A Novel Framework for Multi-class Classification
The discriminator learns to model label dependency by discriminating real and generated label sets.
Improved Audio Embeddings by Adjacency-Based Clustering with Applications in Spoken Term Detection
Embedding audio signal segments into vectors with fixed dimensionality is attractive because all following processing will be easier and more efficient, for example modeling, classifying or indexing.
Code-switching Sentence Generation by Generative Adversarial Networks and its Application to Data Augmentation
To mitigate the issue without expensive human annotation, we proposed an unsupervised method for code-switching data augmentation.
Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model
In this paper we proposed a novel Adversarial Training (AT) approach for end-to-end speech recognition using a Criticizing Language Model (CLM).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data
This can be learned by aligning a small number of spoken words and the corresponding text words in the embedding spaces.
Generative Adversarial Networks for Unpaired Voice Transformation on Impaired Speech
This paper focuses on using voice conversion (VC) to improve the speech intelligibility of surgical patients who have had parts of their articulators removed.
Learning to Encode Text as Human-Readable Summaries using Generative Adversarial Networks
The generator encodes the input text into a shorter word sequence, and the reconstructor recovers the generator input from the generator output.
TopicGAN: Unsupervised Text Generation from Explainable Latent Topics
Learning discrete representations of data and then generating data from the discovered representations have been increasingly studied because the obtained discrete representations can benefit unsupervised learning.
Temporal Pattern Attention for Multivariate Time Series Forecasting
To obtain accurate prediction, it is crucial to model long-term dependency in time series data, which can be achieved to some good extent by recurrent neural network (RNN) with attention mechanism.
Ranked #2 on
Univariate Time Series Forecasting
on Electricity
Proximal Policy Optimization and its Dynamic Version for Sequence Generation
In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning.
Improving Conditional Sequence Generative Adversarial Networks by Stepwise Evaluation
To stabilize the training of SeqGAN, Monte Carlo tree search (MCTS) or reward at every generation step (REGS) is used to evaluate the goodness of a generated subsequence.
Towards Audio to Scene Image Synthesis using Generative Adversarial Network
Humans can imagine a scene from a sound.
Rhythm-Flexible Voice Conversion without Parallel Data Using Cycle-GAN over Phoneme Posteriorgram Sequences
In this way, the length constraint mentioned above is removed to offer rhythm-flexible voice conversion without requiring parallel data.
Sound Audio and Speech Processing
ODSQA: Open-domain Spoken Question Answering Dataset
Reading comprehension by machine has been widely studied, but machine comprehension of spoken content is still a less investigated problem.
Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection
In this paper, we extend audio Word2Vec from word-level to utterance-level by proposing a new segmental audio Word2Vec, in which unsupervised spoken word boundary segmentation and audio Word2Vec are jointly learned and mutually enhanced, so an utterance can be directly represented as a sequence of vectors carrying phonetic structure information.
