Search Results for author:
Found 56 papers, 27 papers with code
Self-supervised Representation Learning for Speech Processing
Due to the growing popularity of SSL, and the shared mission of the areas in bringing speech and language technologies to more use cases with better quality and scaling the technologies for under-represented languages, we propose this tutorial to systematically survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing.
Disentangled Training with Adversarial Examples For Robust Small-footprint Keyword Spotting
A keyword spotting (KWS) engine that is continuously running on device is exposed to various speech signals that are usually unseen before.
SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks
Motivated by the strengths of prompting, we are the first to explore the potential of prompting speech LMs in the domain of speech processing.
Text Quality-Based Pruning for Efficient Training of Language Models
In recent times training Language Models (LMs) have relied on computationally heavy training over massive datasets which makes this training process extremely laborious.
MoDE: CLIP Data Experts via Clustering
The success of contrastive language-image pretraining (CLIP) relies on the supervision from the pairing between images and captions, which tends to be noisy in web-crawled data.
A Large-Scale Evaluation of Speech Foundation Models
In this work, we establish the Speech processing Universal PERformance Benchmark (SUPERB) to study the effectiveness of the paradigm for speech.
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
We introduce VoiceCraft, a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts.
SpeechDPR: End-to-End Spoken Passage Retrieval for Open-Domain Spoken Question Answering
However, the real-world problem of Open-domain SQA (openSQA), in which the machine needs to first retrieve passages that possibly contain the answer from a spoken archive in addition, was never considered.
GSQA: An End-to-End Model for Generative Spoken Question Answering
Furthermore, the GSQA model has only been fine-tuned on the spoken extractive QA dataset.
FLAP: Fast Language-Audio Pre-training
We propose Fast Language-Audio Pre-training (FLAP), a self-supervised approach that efficiently and effectively learns aligned audio and language representations through masking, contrastive learning and reconstruction.
Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks
Despite the success of ICL in NLP, little work is exploring the possibility of ICL in speech processing.
SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT
Data-driven unit discovery in self-supervised learning (SSL) of speech has embarked on a new era of spoken language processing.
Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond
The 2023 Multilingual Speech Universal Performance Benchmark (ML-SUPERB) Challenge expands upon the acclaimed SUPERB framework, emphasizing self-supervised models in multilingual speech recognition and language identification.
Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Model
Notably, in the low-resource scenario, prompting consistently outperforms adapter tuning.
Demystifying CLIP Data
We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective.
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information.
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs.
Ranked #2 on
Text-to-Image Generation
on MS COCO
Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target
However, the absence of intermediate targets and training guidance for textless SLU often results in suboptimal performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering
We propose EAR, a query Expansion And Reranking approach for improving passage retrieval, with the application to open-domain question answering.
Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model
In this paper, we show that representations capturing syllabic units emerge when training a self-supervised speech model with a visually-grounded training objective.
ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks.
DINOv2: Learning Robust Visual Features without Supervision
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision.
Ranked #1 on
Image Retrieval
on AmsterTime
(using extra training data)
SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks
For speech processing, SpeechPrompt shows its high parameter efficiency and competitive performance on a few speech classification tasks.
Ranked #17 on
Spoken Language Understanding
on Fluent Speech Commands
(using extra training data)
MAViL: Masked Audio-Video Learners
We present Masked Audio-Video Learners (MAViL) to train audio-visual representations.
Introducing Semantics into Speech Encoders
Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+10
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We present the SUPERB challenge at SLT 2022, which aims at learning self-supervised speech representation for better performance, generalization, and efficiency.
Exploring Efficient-tuning Methods in Self-supervised Speech Models
In downstream tasks, the parameters of SSL models are frozen, and only the adapters are trained.
Self-Supervised Speech Representation Learning: A Review
Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Meta Learning for Natural Language Processing: A Survey
Deep learning has been the mainstream technique in natural language processing (NLP) area.
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings.
Ranked #13 on
Semantic Textual Similarity
on STS16
SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks
We report in this paper the first exploration of the prompt tuning paradigm for speech processing tasks based on Generative Spoken Language Model (GSLM).
Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognition
Although deep learning-based end-to-end Automatic Speech Recognition (ASR) has shown remarkable performance in recent years, it suffers severe performance regression on test samples drawn from different data distributions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities
In this paper, we introduce SUPERB-SG, a new benchmark focused on evaluating the semantic and generative capabilities of pre-trained models by increasing task diversity and difficulty over SUPERB.
DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering
We empirically showed that DUAL yields results comparable to those obtained by cascading ASR and text QA model and robust to real-world data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
QaNER: Prompting Question Answering Models for Few-shot Named Entity Recognition
Recently, prompt-based learning for pre-trained language models has succeeded in few-shot Named Entity Recognition (NER) by exploiting prompts as task guidance to increase label efficiency.
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora
We evaluate PTLM's ability to adapt to new corpora while retaining learned knowledge in earlier corpora.
Pairwise Supervised Contrastive Learning of Sentence Representations
Many recent successes in sentence representation learning have been achieved by simply fine-tuning on the Natural Language Inference (NLI) datasets with triplet loss or siamese loss.
Meta Learning and Its Applications to Natural Language Processing
Meta-learning is one of the most important new techniques in machine learning in recent years.
Mitigating Biases in Toxic Language Detection through Invariant Rationalization
Automatic detection of toxic language plays an essential role in protecting social media users, especially minority groups, from verbal abuse.
Meta-learning for downstream aware and agnostic pretraining
Neural network pretraining is gaining attention due to its outstanding performance in natural language processing applications.
SUPERB: Speech processing Universal PERformance Benchmark
SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data.
Cooperative Self-training of Machine Reading Comprehension
Pretrained language models have significantly improved the performance of downstream language understanding tasks, including extractive question answering, by providing high-quality contextualized word embeddings.
Ranked #1 on
Question Answering
on MRQA out-of-domain
Extractive Question-Answering
Machine Reading Comprehension
+6
Knowledge Grounded Conversational Symptom Detection with Graph Memory Networks
Given a set of explicit symptoms provided by the patient to initiate a dialog for diagnosing, the system is trained to collect implicit symptoms by asking questions, in order to collect more information for making an accurate diagnosis.
Zero-shot Generalization in Dialog State Tracking through Generative Question Answering
Dialog State Tracking (DST), an integral part of modern dialog systems, aims to track user preferences and constraints (slots) in task-oriented dialogs.
Educational Content Linking for Enhancing Learning Need Remediation in MOOCs
By linking and organizing pieces of learning content scattered in various course materials into an easily accessible structure, we hypothesize that this framework can provide learners guidance and improve content navigation.
Meta learning to classify intent and slot labels with noisy few shot examples
Recently deep learning has dominated many machine learning areas, including spoken language understanding (SLU).
Towards Semi-Supervised Semantics Understanding from Speech
Much recent work on Spoken Language Understanding (SLU) falls short in at least one of three ways: models were trained on oracle text input and neglected the Automatics Speech Recognition (ASR) outputs, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data.
Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data.
Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding
The light encoder architecture separates the shared pre-trained networks from the mappings of generally encoded knowledge to specific domains of SLU, allowing for the domain adaptation to be performed solely at the light encoder and thus increasing efficiency.
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
We present a large-scale comparison of various self-supervised models.
Prototypical Q Networks for Automatic Conversational Diagnosis and Few-Shot New Disease Adaption
Experiments showed that the ProtoQN significantly outperformed the baseline DQN model in both supervised and few-shot learning scenarios, and achieves state-of-the-art few-shot learning performances.
Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
We use the representations with two downstream tasks, speaker identification, and phoneme classification.
Learning Robust Dialog Policies in Noisy Environments
In this paper, we focus on learning robust dialog policies to recover from these errors.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A Coarse-to-Fine Indoor Layout Estimation (CFILE) Method
The task of estimating the spatial layout of cluttered indoor scenes from a single RGB image is addressed in this work.
GAL: A Global-Attributes Assisted Labeling System for Outdoor Scenes
The proposed Global-attributes Assisted Labeling (GAL) system exploits both local features and global attributes.
Measuring and Predicting Tag Importance for Image Retrieval
Textual data such as tags, sentence descriptions are combined with visual cues to reduce the semantic gap for image retrieval applications in today's Multimodal Image Retrieval (MIR) systems.
