Search Results for author:
Found 229 papers, 91 papers with code
EURO: ESPnet Unsupervised ASR Open-source Toolkit
This paper describes the ESPnet Unsupervised ASR Open-source Toolkit (EURO), an end-to-end open-source toolkit for unsupervised automatic speech recognition (UASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech.
Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
We use the representations with two downstream tasks, speaker identification, and phoneme classification.
Defense for Black-box Attacks on Anti-spoofing Models by Self-Supervised Learning
To explore this issue, we proposed to employ Mockingjay, a self-supervised learning based model, to protect anti-spoofing models against adversarial attacks in the black-box scenario.
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
We present a large-scale comparison of various self-supervised models.
Utilizing Self-supervised Representations for MOS Prediction
In this paper, we use self-supervised pre-trained models for MOS prediction.
S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations
AUTOVC used dvector to extract speaker information, and self-supervised learning (SSL) features like wav2vec 2. 0 is used in FragmentVC to extract the phonetic content information.
DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT
Self-supervised speech representation learning methods like wav2vec 2. 0 and Hidden-unit BERT (HuBERT) leverage unlabeled speech data for pre-training and offer good representations for numerous speech processing tasks.
SUPERB: Speech processing Universal PERformance Benchmark
SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data.
S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations
In this work, we provide a series of in-depth analyses by benchmarking on the two tasks in VCC2020, namely intra-/cross-lingual any-to-one (A2O) VC, as well as an any-to-any (A2A) setting.
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities
In this paper, we introduce SUPERB-SG, a new benchmark focused on evaluating the semantic and generative capabilities of pre-trained models by increasing task diversity and difficulty over SUPERB.
Self-supervised Representation Learning for Speech Processing
Due to the growing popularity of SSL, and the shared mission of the areas in bringing speech and language technologies to more use cases with better quality and scaling the technologies for under-represented languages, we propose this tutorial to systematically survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing.
Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis
Prosody modeling is an essential component in modern text-to-speech (TTS) frameworks.
Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech
The few-shot multi-speaker multi-style voice cloning task is to synthesize utterances with voice and speaking style similar to a reference speaker given only a few reference samples.
Sequence-to-sequence Automatic Speech Recognition with Word Embedding Regularization and Fused Decoding
This allows the decoder to consider the semantic consistency during decoding by absorbing the information carried by the transformed decoder feature, which is learned to be close to the target word embedding.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Temporal Pattern Attention for Multivariate Time Series Forecasting
To obtain accurate prediction, it is crucial to model long-term dependency in time series data, which can be achieved to some good extent by recurrent neural network (RNN) with attention mechanism.
Ranked #2 on
Univariate Time Series Forecasting
on Electricity
One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
Recently, voice conversion (VC) without parallel data has been successfully adapted to multi-target scenario in which a single model is trained to convert the input voice to many different speakers.
Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations
The decoder then takes the speaker-independent latent representation and the target speaker embedding as the input to generate the voice of the target speaker with the linguistic content of the source utterance.
Tree Transformer: Integrating Tree Structures into Self-Attention
This paper proposes Tree Transformer, which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures.
FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention
Any-to-any voice conversion aims to convert the voice from and to any speakers even unseen during training, which is much more challenging compared to one-to-one or many-to-many tasks, but much more attractive in real-world scenarios.
Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
On the one hand, speaker adaptation methods fine-tune a trained multi-speaker text-to-speech (TTS) model with few enrolled samples.
Codec-SUPERB: An In-Depth Analysis of Sound Codec Models
The sound codec's dual roles in minimizing data transmission latency and serving as tokenizers underscore its critical importance.
Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion
We found that the proposed encoding method offers automatic extraction of speech content from speaker style, and is sufficient to cover full linguistic content in a given language.
AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization
With a proper activation as an information bottleneck on content embeddings, the trade-off between the synthesis quality and the speaker similarity of the converted speech is improved drastically.
Audio and Speech Processing Sound
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly.
SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks
We report in this paper the first exploration of the prompt tuning paradigm for speech processing tasks based on Generative Spoken Language Model (GSLM).
Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
To achieve comprehensive coverage of diverse speech tasks and harness instruction tuning, we invite the community to collaborate and contribute, facilitating the dynamic growth of the benchmark.
LAMOL: LAnguage MOdeling for Lifelong Language Learning
We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling.
Ranked #4 on
Continual Learning
on ASC (19 tasks)
Adversarial Sample Detection for Speaker Verification by Neural Vocoders
We also show that the neural vocoder adopted in the detection framework is dataset-independent.
VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture
Voice conversion (VC) is a task that transforms the source speaker's timbre, accent, and tones in audio into another one's while preserving the linguistic content.
WG-WaveNet: Real-Time High-Fidelity Speech Synthesis without GPU
As we design a flow-based model that is heavily compressed, the proposed model requires much less computational resources compared to other waveform generation models during both training and inference time; even though the model is highly compressed, the post-filter maintains the quality of generated waveform.
Speech Synthesis
Text-To-Speech Synthesis
Audio and Speech Processing
Sound
Learning to Encode Text as Human-Readable Summaries using Generative Adversarial Networks
The generator encodes the input text into a shorter word sequence, and the reconstructor recovers the generator input from the generator output.
Stabilizing Label Assignment for Speech Separation by Self-supervised Pre-training
Speech separation has been well developed, with the very successful permutation invariant training (PIT) approach, although the frequent label assignment switching happening during PIT training remains to be a problem when better convergence speed and achievable performance are desired.
Ranked #6 on
Speech Separation
on Libri2Mix
(using extra training data)
ODSQA: Open-domain Spoken Question Answering Dataset
Reading comprehension by machine has been widely studied, but machine comprehension of spoken content is still a less investigated problem.
Multimodal Transformer Distillation for Audio-Visual Synchronization
This paper proposed an MTDVocaLiST model, which is trained by our proposed multimodal Transformer distillation (MTD) loss.
MelHuBERT: A simplified HuBERT on Mel spectrograms
Self-supervised models have had great success in learning speech representations that can generalize to various downstream tasks.
DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphs
This paper proposes a new task about how to apply dynamic knowledge graphs in neural conversation model and presents a novel TV series conversation corpus (DyKgChat) for the task.
Adversarial Attacks on Spoofing Countermeasures of automatic speaker verification
High-performance spoofing countermeasure systems for automatic speaker verification (ASV) have been proposed in the ASVspoof 2019 challenge.
Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension
Reading comprehension has been widely studied.
Ranked #4 on
Spoken Language Understanding
on Spoken-SQuAD
Mitigating the Impact of Speech Recognition Errors on Spoken Question Answering by Adversarial Domain Adaptation
Spoken question answering (SQA) is challenging due to complex reasoning on top of the spoken documents.
Ranked #3 on
Spoken Language Understanding
on Spoken-SQuAD
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Defending Your Voice: Adversarial Attack on Voice Conversion
We introduce human imperceptible noise into the utterances of a speaker whose voice is to be defended.
Improving Conditional Sequence Generative Adversarial Networks by Stepwise Evaluation
To stabilize the training of SeqGAN, Monte Carlo tree search (MCTS) or reward at every generation step (REGS) is used to evaluate the goodness of a generated subsequence.
DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering
We empirically showed that DUAL yields results comparable to those obtained by cascading ASR and text QA model and robust to real-world data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information.
Query-based Attention CNN for Text Similarity Map
This network is composed of compare mechanism, two-staged CNN architecture with attention mechanism, and a prediction layer.
Learning Chinese Word Representations From Glyphs Of Characters
The character glyph features are directly learned from the bitmaps of characters by convolutional auto-encoder(convAE), and the glyph features improve Chinese word representations which are already enhanced by character embeddings.
TaylorGAN: Neighbor-Augmented Policy Update Towards Sample-Efficient Natural Language Generation
Score function-based natural language generation (NLG) approaches such as REINFORCE, in general, suffer from low sample efficiency and training instability problems.
TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation
Score function-based natural language generation (NLG) approaches such as REINFORCE, in general, suffer from low sample efficiency and training instability problems.
On the Utility of Self-supervised Models for Prosody-related Tasks
We find that 13 of the 15 SSL models outperformed the baseline on all the prosody-related tasks.
Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognition
Although deep learning-based end-to-end Automatic Speech Recognition (ASR) has shown remarkable performance in recent years, it suffers severe performance regression on test samples drawn from different data distributions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Improved Speech Separation with Time-and-Frequency Cross-domain Joint Embedding and Clustering
Substantial effort has been reported based on approaches over spectrogram, which is well known as the standard time-and-frequency cross-domain representation for speech signals.
Ranked #24 on
Speech Separation
on WSJ0-2mix
Interrupted and cascaded permutation invariant training for speech separation
Permutation Invariant Training (PIT) has long been a stepping stone method for training speech separation model in handling the label ambiguity problem.
Ranked #22 on
Speech Separation
on WSJ0-2mix
Noise Adaptive Speech Enhancement using Domain Adversarial Training
The proposed noise adaptive SE system contains an encoder-decoder-based enhancement model and a domain discriminator model.
Sound Audio and Speech Processing
Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation
Simultaneous speech translation (SimulST) is a challenging task aiming to translate streaming speech before the complete input is observed.
Generative Adversarial Networks for Unpaired Voice Transformation on Impaired Speech
This paper focuses on using voice conversion (VC) to improve the speech intelligibility of surgical patients who have had parts of their articulators removed.
Order-free Learning Alleviating Exposure Bias in Multi-label Classification
In this paper, we propose a new framework for MLC which does not rely on a predefined label order and thus alleviates exposure bias.
SpeechNet: A Universal Modularized Model for Speech Processing Tasks
The multi-task learning of a wide variety of speech processing tasks with a universal model has not been studied.
T5lephone: Bridging Speech and Text Self-supervised Models for Spoken Language Understanding via Phoneme level T5
In Spoken language understanding (SLU), a natural solution is concatenating pre-trained speech models (e. g. HuBERT) and pretrained language models (PLM, e. g. T5).
AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks
Transformer-based pre-trained models with millions of parameters require large storage.
EMO-SUPERB: An In-depth Look at Speech Emotion Recognition
Speech emotion recognition (SER) is a pivotal technology for human-computer interaction systems.
A Closer Look into Automatic Evaluation Using Large Language Models
In this paper, we analyze LLM evaluation (Chiang and Lee, 2023) and G-Eval (Liu et al., 2023), and we discuss how those details in the evaluation process change how well the ratings given by LLMs correlate with human ratings.
Auto-KWS 2021 Challenge: Task, Datasets, and Baselines
Auto-KWS 2021 challenge calls for automated machine learning (AutoML) solutions to automate the process of applying machine learning to a customized keyword spotting task.
Pretrained Language Model Embryology: The Birth of ALBERT
These findings suggest that knowledge of a pretrained model varies during pretraining, and having more pretrain steps does not necessarily provide a model with more comprehensive knowledge.
Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data.
The Efficacy of Self-Supervised Speech Models for Audio Representations
Extensive experiments on speech and non-speech audio datasets are conducted to investigate the representation abilities of our ensemble method and its single constituent model.
J-Net: Randomly weighted U-Net for audio source separation
According to these discoveries, we pose two questions: what is the value of randomly weighted networks in difficult generative audio tasks such as audio source separation and does such positive correlation still exist when it comes to large random networks and their trained counterparts?
Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation
We study the possibilities of building a non-autoregressive speech-to-text translation model using connectionist temporal classification (CTC), and use CTC-based automatic speech recognition as an auxiliary task to improve the performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Membership Inference Attacks Against Self-supervised Speech Models
Recently, adapting the idea of self-supervised learning (SSL) on continuous speech has started gaining attention.
Anticipation-Free Training for Simultaneous Machine Translation
Existing methods increase latency or introduce adaptive read-write policies for SimulMT models to handle local reordering and improve translation quality.
Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder
The vector representations of fixed dimensionality for words (in text) offered by Word2Vec have been shown to be very useful in many application scenarios, in particular due to the semantic information they carry.
Voting for the right answer: Adversarial defense for speaker verification
Automatic speaker verification (ASV) is a well developed technology for biometric identification, and has been ubiquitous implemented in security-critic applications, such as banking and access control.
Unsupervised Multiple Choices Question Answering: Start Learning from Basic Knowledge
In this paper, we study the possibility of almost unsupervised Multiple Choices Question Answering (MCQA).
Universal Adaptor: Converting Mel-Spectrograms Between Different Configurations for Speech Synthesis
Most recent speech synthesis systems are composed of a synthesizer and a vocoder.
Over-Reasoning and Redundant Calculation of Large Language Models
Large language models (LLMs) can solve problems step-by-step.
Gate Activation Signal Analysis for Gated Recurrent Neural Networks and Its Correlation with Phoneme Boundaries
In this paper we analyze the gate activation signals inside the gated recurrent neural networks, and find the temporal structure of such signals is highly correlated with the phoneme boundaries.
Rhythm-Flexible Voice Conversion without Parallel Data Using Cycle-GAN over Phoneme Posteriorgram Sequences
In this way, the length constraint mentioned above is removed to offer rhythm-flexible voice conversion without requiring parallel data.
Sound Audio and Speech Processing
Mitigating Biases in Toxic Language Detection through Invariant Rationalization
Automatic detection of toxic language plays an essential role in protecting social media users, especially minority groups, from verbal abuse.
How to Estimate Model Transferability of Pre-Trained Speech Models?
In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks.
Understanding Self-Attention of Self-Supervised Audio Transformers
Self-supervised Audio Transformers (SAT) enable great success in many downstream speech applications like ASR, but how they work has not been widely explored yet.
Improving generalizability of distilled self-supervised speech processing models under distorted settings
Self-supervised learned (SSL) speech pre-trained models perform well across various speech processing tasks.
Cascading and Direct Approaches to Unsupervised Constituency Parsing on Spoken Sentences
The goal is to determine the spoken sentences' hierarchical syntactic structure in the form of constituency parse trees, such that each node is a span of audio that corresponds to a constituent.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Models
In this paper, we introduce MiniSUPERB, a lightweight benchmark that efficiently evaluates SSL speech models with comparable results to SUPERB but lower computational costs significantly.
Why We Should Report the Details in Subjective Evaluation of TTS More Rigorously
This paper emphasizes the importance of reporting experiment details in subjective evaluations and demonstrates how such details can significantly impact evaluation results in the field of speech synthesis.
What does a network layer hear? Analyzing hidden representations of end-to-end ASR through speech synthesis
End-to-end speech recognition systems have achieved competitive results compared to traditional systems.
Looking for Clues of Language in Multilingual BERT to Improve Cross-lingual Generalization
Token embeddings in multilingual BERT (m-BERT) contain both language and semantic information.
On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets
In this work, we study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
Compressing Transformer-based self-supervised models for speech processing
Despite the success of Transformers in self- supervised learning with applications to various downstream tasks, the computational cost of training and inference remains a major challenge for applying these models to a wide spectrum of devices.
Code-switching Sentence Generation by Generative Adversarial Networks and its Application to Data Augmentation
To mitigate the issue without expensive human annotation, we proposed an unsupervised method for code-switching data augmentation.
Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languages
We introduce a new zero resource code-switched speech benchmark designed to directly assess the code-switching capabilities of self-supervised speech encoders.
Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations
We show that LLMs can generate paragraphs that contain verifiable facts, but the facts are combined to form a non-factual paragraph due to entity ambiguity.
Towards Unsupervised Automatic Speech Recognition Trained by Unaligned Speech and Text only
In this work, we propose a framework to achieve unsupervised ASR on a read English speech dataset, where audio and text are unaligned.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Query-by-example Spoken Term Detection using Attention-based Multi-hop Networks
Retrieving spoken content with spoken queries, or query-by- example spoken term detection (STD), is attractive because it makes possible the matching of signals directly on the acoustic level without transcribing them into text.
Supervised and Unsupervised Transfer Learning for Question Answering
Although transfer learning has been shown to be successful for tasks like object and speech recognition, its applicability to question answering (QA) has yet to be well-studied.
Scalable Sentiment for Sequence-to-sequence Chatbot Response with Performance Analysis
Conventional seq2seq chatbot models only try to find the sentences with the highest probabilities conditioned on the input sequences, without considering the sentiment of the output sentences.
Joint Learning of Interactive Spoken Content Retrieval and Trainable User Simulator
User-machine interaction is crucial for information retrieval, especially for spoken content retrieval, because spoken content is difficult to browse, and speech recognition has a high degree of uncertainty.
Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings
Unsupervised discovery of acoustic tokens from audio corpora without annotation and learning vector representations for these tokens have been widely studied.
Mitigating the Impact of Speech Recognition Errors on Chatbot using Sequence-to-Sequence Model
We apply sequence-to-sequence model to mitigate the impact of speech recognition errors on open domain end-to-end dialog generation.
Order-Preserving Abstractive Summarization for Spoken Content Based on Connectionist Temporal Classification
Connectionist temporal classification (CTC) is a powerful approach for sequence-to-sequence learning, and has been popularly used in speech recognition.
Personalized word representations Carrying Personalized Semantics Learned from Social Network Posts
In this framework, universal background word vectors are first learned from the background corpora, and then adapted by the personalized corpus for each individual user to learn the personalized word vectors.
Language Transfer of Audio Word2Vec: Learning Audio Segment Representations without Target Language Data
We train SA from one language (source language) and use it to extract the vector representation of the audio segments of another language (target language).
Hierarchical Attention Model for Improved Machine Comprehension of Spoken Content
Multimedia or spoken content presents more attractive information than plain text content, but the former is more difficult to display on a screen and be selected by a user.
Abstractive Headline Generation for Spoken Content by Attentive Recurrent Neural Networks with ASR Error Modeling
In this way, the model for abstractive headline generation for spoken content can be learned from abundant text data and the ASR data for some recognizers.
Attention-based Memory Selection Recurrent Network for Language Modeling
Recurrent neural networks (RNNs) have achieved great success in language modeling.
Personalizing Universal Recurrent Neural Network Language Model with User Characteristic Features by Social Network Crowdsouring
With the popularity of mobile devices, personalized speech recognizer becomes more realizable today and highly attractive.
Interactive Spoken Content Retrieval by Deep Reinforcement Learning
In our previous work, some hand-crafted states estimated from the present retrieval results are used to determine the proper actions.
Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machine
Multimedia or spoken content presents more attractive information than plain text content, but it's more difficult to display on a screen and be selected by a user.
Neural Attention Models for Sequence Classification: Analysis and Application to Key Term Extraction and Dialogue Act Detection
The major difficulty of sequence labeling is that when the input sequence is long, it can include many noisy or irrelevant part.
An Iterative Deep Learning Framework for Unsupervised Discovery of Speech Features and Linguistic Units with Applications on Spoken Term Detection
The multiple sets of token labels are then used as the targets of a Multi-target Deep Neural Network (MDNN) trained on low-level acoustic features.
Towards Structured Deep Neural Network for Automatic Speech Recognition
In this paper we propose the Structured Deep Neural Network (structured DNN) as a structured and deep learning framework.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A Multi-layered Acoustic Tokenizing Deep Neural Network (MAT-DNN) for Unsupervised Discovery of Linguistic Units and Generation of High Quality Features
The Multi-layered Acoustic Tokenizer (MAT) proposed in this work automatically discovers multiple sets of acoustic tokens from the given corpus.
Towards Structured Deep Neural Network for Automatic Speech Recognition
In this paper we propose the Structured Deep Neural Network (Structured DNN) as a structured and deep learning algorithm, learning to find the best structured object (such as a label sequence) given a structured input (such as a vector sequence) by globally considering the mapping relationships between the structure rather than item by item.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval
Stage 1 performs phonetic embedding with speaker characteristics disentangled.
Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection
In this paper, we extend audio Word2Vec from word-level to utterance-level by proposing a new segmental audio Word2Vec, in which unsupervised spoken word boundary segmentation and audio Word2Vec are jointly learned and mutually enhanced, so an utterance can be directly represented as a sequence of vectors carrying phonetic structure information.
Towards Audio to Scene Image Synthesis using Generative Adversarial Network
Humans can imagine a scene from a sound.
Proximal Policy Optimization and its Dynamic Version for Sequence Generation
In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning.
Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data
This can be learned by aligning a small number of spoken words and the corresponding text words in the embedding spaces.
Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model
In this paper we proposed a novel Adversarial Training (AT) approach for end-to-end speech recognition using a Criticizing Language Model (CLM).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Improved Audio Embeddings by Adjacency-Based Clustering with Applications in Spoken Term Detection
Embedding audio signal segments into vectors with fixed dimensionality is attractive because all following processing will be easier and more efficient, for example modeling, classifying or indexing.
Adversarial Learning of Label Dependency: A Novel Framework for Multi-class Classification
The discriminator learns to model label dependency by discriminating real and generated label sets.
Completely Unsupervised Speech Recognition By A Generative Adversarial Network Harmonized With Iteratively Refined Hidden Markov Models
Producing a large annotated speech corpus for training ASR systems remains difficult for more than 95% of languages all over the world which are low-resourced, but collecting a relatively big unlabeled data set for such languages is more achievable.
From Semi-supervised to Almost-unsupervised Speech Recognition with Very-low Resource by Jointly Learning Phonetic Structures from Audio and Text Embeddings
However, we note human babies start to learn the language by the sounds (or phonetic structures) of a small number of exemplar words, and "generalize" such knowledge to other words without hearing a large amount of data.
End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning
In this paper, we aim to build TTS systems for such low-resource (target) languages where only very limited paired data are available.
Cross-Lingual Transfer Learning for Question Answering
In this paper, we explore the problem of cross-lingual transfer learning for QA, where a source language task with plentiful annotations is utilized to improve the performance of a QA model on a target language task with limited available annotations.
Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets
Paraphrase generation is an interesting and challenging NLP task which has numerous practical applications.
Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
Because it is not feasible to collect training data for every language, there is a growing interest in cross-lingual transfer learning.
SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
In addition to the potential of end-to-end SQA, the SpeechBERT can also be considered for many other spoken language understanding tasks just as BERT for many text processing tasks.
Ranked #3 on
Spoken Language Understanding
on Spoken-SQuAD
Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning
In this paper we propose a Sequential Representation Quantization AutoEncoder (SeqRQ-AE) to learn from primarily unpaired audio data and produce sequences of representations very close to phoneme sequences of speech utterances.
Meta Learning for End-to-End Low-Resource Speech Recognition
In this paper, we proposed to apply meta learning approach for low-resource automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Training a code-switching language model with monolingual data
A lack of code-switching data complicates the training of code-switching (CS) language models.
Towards Robust Neural Vocoding for Speech Generation: A Survey
We found out that the speaker variety is much more important for achieving a universal vocoder than the language.
MITAS: A Compressed Time-Domain Audio Separation Network with Parameter Sharing
Our experimental results first confirmed the robustness of our MiTAS on two types of perturbations in mixed audio.
BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
Although Bidirectional Encoder Representations from Transformers (BERT) have achieved tremendous success in many natural language processing (NLP) tasks, it remains a black box.
Defense against adversarial attacks on spoofing countermeasures of ASV
Various forefront countermeasure methods for automatic speaker verification (ASV) with considerable performance in anti-spoofing are proposed in the ASVspoof 2019 challenge.
A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings.
Learning Interpretable and Discrete Representations with Adversarial Training for Unsupervised Text Classification
Also, the performance is on par with a recently proposed weakly-supervised text classification method.
End-to-end Whispered Speech Recognition with Frequency-weighted Approaches and Pseudo Whisper Pre-training
Whispering is an important mode of human speech, but no end-to-end recognition results for it were reported yet, probably due to the scarcity of available whispered speech data.
DARTS-ASR: Differentiable Architecture Search for Multilingual Speech Recognition and Adaptation
In order to examine the generalizability of DARTS-ASR, we apply our approach not only on many languages to perform monolingual ASR, but also on a multilingual ASR setting.
Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation
The experiment results demonstrate that with only an hour of paired speech data, no matter the paired data is from multiple speakers or a single speaker, the proposed model can generate intelligible speech in different voices.
Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation
Speech translation (ST) aims to learn transformations from speech in the source language to the text in the target language.
Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers
In this paper, we seek solutions for reducing the computation complexity of transformer-based models for speech representation learning.
Investigation of Sentiment Controllable Chatbot
Conventional seq2seq chatbot models attempt only to find sentences with the highest probabilities conditioned on the input sequences, without considering the sentiment of the output sentences.
What makes multilingual BERT multilingual?
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings.
Pre-Training a Language Model Without Human Language
In this paper, we study how the intrinsic nature of pre-training data contributes to the fine-tuned downstream performance.
Adversarial defense for automatic speaker verification by cascaded self-supervised learning models
Automatic speaker verification (ASV) is one of the core technologies in biometric identification.
Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability
This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications.
Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention
Our framework includes a guiding chatbot and an interlocutor model that plays the role of humans.
Towards Lifelong Learning of End-to-end ASR
We can collect new data describing the new environment and fine-tune the system, but this naturally leads to higher error rates for the earlier datasets, referred to as catastrophic forgetting.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Non-autoregressive Mandarin-English Code-switching Speech Recognition
Mandarin-English code-switching (CS) is frequently used among East and Southeast Asian people.
How Far Are We from Robust Voice Conversion: A Survey
Voice conversion technologies have been greatly improved in recent years with the help of deep learning, but their capabilities of producing natural sounding utterances in different conditions remain unclear.
Improving Cross-Lingual Reading Comprehension with Self-Training
Substantial improvements have been made in machine reading comprehension, where the machine answers questions based on a given context.
Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning
This work is among the first to perform adversarial defense for ASV without knowing the specific attack algorithms.
Parallelized Reverse Curriculum Generation
For reinforcement learning (RL), it is challenging for an agent to master a task that requires a specific series of actions due to sparse rewards.
Meta Learning and Its Applications to Natural Language Processing
Meta-learning is one of the most important new techniques in machine learning in recent years.
Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models
Code-switching (CS) is common in daily conversations where more than one language is used within a sentence.
Analyzing the Robustness of Unsupervised Speech Recognition
In this work, we further analyze the training robustness of unsupervised ASR on the domain mismatch scenarios in which the domains of unpaired speech and text are different.
CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning
Apart from the coherence and fluency of responses, an empathetic chatbot emphasizes more on people's feelings.
An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Toward Degradation-Robust Voice Conversion
However, in real-world scenarios, it is difficult to collect clean utterances of a speaker, and they are usually degraded by noises or reverberations.
Don't speak too fast: The impact of data bias on self-supervised speech models
Self-supervised Speech Models (S3Ms) have been proven successful in many speech downstream tasks, like ASR.
Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuning
It has been shown that an SSL pretraining model can achieve excellent performance in various downstream tasks of speech processing.
Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models’ Transferability
This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications.
Characterizing the adversarial vulnerability of speech self-supervised learning
As the paradigm of the self-supervised learning upstream model followed by downstream tasks arouses more attention in the speech community, characterizing the adversarial robustness of such paradigm is of high priority.
TopicGAN: Unsupervised Text Generation from Explainable Latent Topics
Learning discrete representations of data and then generating data from the discovered representations have been increasingly studied because the obtained discrete representations can benefit unsupervised learning.
Partially Fake Audio Detection by Self-attention-based Fake Span Discovery
Also ADD 2022 is the first challenge to propose the partially fake audio detection task.
Spoofing-Aware Speaker Verification by Multi-Level Fusion
In the second-level fusion, the CM score and ASV scores directly from ASV systems will be concatenated into a prediction block for the final decision.
Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation
Speech distortions are a long-standing problem that degrades the performance of supervisely trained speech processing models.
On the Efficiency of Integrating Self-supervised Learning and Meta-learning for User-defined Few-shot Keyword Spotting
User-defined keyword spotting is a task to detect new spoken terms defined by users.
DDOS: A MOS Prediction Framework utilizing Domain Adaptive Pre-training and Distribution of Opinion Scores
Mean opinion score (MOS) is a typical subjective evaluation metric for speech synthesis systems.
Re-Examining Human Annotations for Interpretable NLP
Our results reveal that the annotation quality is highly subject to the workers' qualification, and workers can be guided to provide certain annotations by the instructions.
Understanding, Detecting, and Separating Out-of-Distribution Samples and Adversarial Samples in Text Classification
Based on our observations, we propose a simple method to separate ID, OOD, and Adv samples using the hidden representations and output probabilities of the model.
XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding
Transformer-based models are widely used in natural language understanding (NLU) tasks, and multimodal transformers have been effective in visual-language tasks.
Meta Learning for Natural Language Processing: A Survey
Deep learning has been the mainstream technique in natural language processing (NLP) area.
Silence is Sweeter Than Speech: Self-Supervised Model Using Silence to Store Speaker Information
We found that HuBERT stores speaker information in representations whose positions correspond to silences in a waveform.
Self-Supervised Speech Representation Learning: A Review
Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Structured Prompt Tuning
We propose structured prompt tuning, a simple and effective method to improve prompt tuning.
Searching for the Essence of Adversarial Perturbations
Neural networks have demonstrated state-of-the-art performance in various machine learning fields.
Learning to Generate Prompts for Dialogue Generation through Reinforcement Learning
Much literature has shown that prompt-based learning is an efficient method to make use of the large pre-trained language model.
Tackling Spoofing-Aware Speaker Verification with Multi-Model Fusion
However, previous works show that state-of-the-art ASV models are seriously vulnerable to voice spoofing attacks, and the recently proposed high-performance spoofing countermeasure (CM) models only focus solely on the standalone anti-spoofing tasks, and ignore the subsequent speaker verification process.
Few-Shot Cross-Lingual TTS Using Transferable Phoneme Embedding
This paper studies a transferable phoneme embedding framework that aims to deal with the cross-lingual text-to-speech (TTS) problem under the few-shot setting.
Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network
GAN training is adopted in the first stage to find the mapping relationship between unpaired speech and phone sequence.
Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection
Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications.
Adversarial Robustness
Audio-Visual Active Speaker Detection
Are Synonym Substitution Attacks Really Synonym Substitution Attacks?
In this paper, we explore the following question: Are synonym substitution attacks really synonym substitution attacks (SSAs)?
Exploring Efficient-tuning Methods in Self-supervised Speech Models
In downstream tasks, the parameters of SSL models are frozen, and only the adapters are trained.
On Compressing Sequences for Self-Supervised Speech Models
Subsampling while training self-supervised models not only improves the overall performance on downstream tasks under certain frame rates, but also brings significant speed-up in inference.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We present the SUPERB challenge at SLT 2022, which aims at learning self-supervised speech representation for better performance, generalization, and efficiency.
M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval
This work investigates the use of large-scale, English-only pre-trained models (CLIP and HuBERT) for multilingual image-speech retrieval.
Once-for-All Sequence Compression for Self-Supervised Speech Models
The sequence length along the time axis is often the dominant factor of the computation in speech processing.
Bridging Speech and Textual Pre-trained Models with Unsupervised ASR
To be specific, we propose to use unsupervised automatic speech recognition (ASR) as a connector that bridges different modalities used in speech and textual pre-trained models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Introducing Semantics into Speech Encoders
Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+10
Model Extraction Attack against Self-supervised Speech Models
In the first stage, SSL is conducted on the large-scale unlabeled corpus to pre-train a small speech model.
General Framework for Self-Supervised Model Priming for Parameter-Efficient Fine-tuning
To tackle these issues, we propose a general framework to enhance the few-shot adaptation and cross-domain generalization ability of parameter-efficient methods.
CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Models
However, such efficient tuning techniques only provide adaptation at the transformer layer, but failed to perform adaptation at the feature extractor.
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks
In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape.
Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs
Aiming to produce reinforcement learning (RL) policies that are human-interpretable and can generalize better to novel scenarios, Trivedi et al. (2021) present a method (LEAPS) that first learns a program embedding space to continuously parameterize diverse programs from a pre-generated program dataset, and then searches for a task-solving program in the learned program embedding space when given a task.
Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue
With the power of large pretrained language models, various research works have integrated knowledge into dialogue systems.
Ensemble knowledge distillation of self-supervised speech models
We tried two different aggregation techniques, layerwise-average and layerwise-concatenation, to the representations of different teacher models and found that the former was more effective.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks
For speech processing, SpeechPrompt shows its high parameter efficiency and competitive performance on a few speech classification tasks.
Ranked #17 on
Spoken Language Understanding
on Fluent Speech Commands
(using extra training data)
Can Large Language Models Be an Alternative to Human Evaluations?
We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.
Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation
We use fully unpaired data to train our unsupervised systems and evaluate our results on CoVoST 2 and CVSS.
ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks.
