Search Results for author:
Found 94 papers, 35 papers with code
Self-supervised Representation Learning for Speech Processing
Due to the growing popularity of SSL, and the shared mission of the areas in bringing speech and language technologies to more use cases with better quality and scaling the technologies for under-represented languages, we propose this tutorial to systematically survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing.
On The Landscape of Spoken Language Models: A Comprehensive Survey
The field of spoken language processing is undergoing a shift from training custom-built, task-specific models toward using and optimizing spoken language models (SLMs) which act as universal speech processing systems.
CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition
This is common in domains like speech, where the widely used connectionist temporal classification (CTC) loss scales with input length and varies with linguistic and acoustic properties, leading to spurious differences between group losses.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Chunk-Distilled Language Modeling
We introduce Chunk-Distilled Language Modeling (CD-LM), an approach to text generation that addresses two challenges in current large language models (LLMs): the inefficiency of token-level generation, and the difficulty of adapting to new data and knowledge.
SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction
Sign language processing has traditionally relied on task-specific models, limiting the potential for transfer learning across tasks.
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
We present Dynamic-SUPERB Phase-2, an open and evolving benchmark for the comprehensive evaluation of instruction-based universal speech models.
Approaching Deep Learning through the Spectral Dynamics of Weights
We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phenomena in deep learning.
Towards Robust Speech Representation Learning for Thousands of Languages
We propose XEUS, a Cross-lingual Encoder for Universal Speech, trained on over 1 million hours of data across 4057 languages, extending the language coverage of SSL models 4-fold.
On the Evaluation of Speech Foundation Models for Spoken Language Understanding
To answer this, we perform an extensive evaluation of multiple supervised and self-supervised SFMs using several evaluation protocols: (i) frozen SFMs with a lightweight prediction head, (ii) frozen SFMs with a complex prediction head, and (iii) fine-tuned SFMs with a lightweight prediction head.
On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models
The Open Whisper-style Speech Model (OWSM) series was introduced to achieve full transparency in building advanced speech-to-text (S2T) foundation models.
DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding
This integration requires the use of a speech encoder, a speech adapter, and an LLM, trained on diverse tasks.
Self-Supervised Speech Representations are More Phonetic than Semantic
Self-supervised speech models (S3Ms) have become an effective backbone for speech applications.
ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets
This paper presents ML-SUPERB~2. 0, which is a new benchmark for evaluating pre-trained SSL and supervised speech models across downstream models, fine-tuning setups, and efficient model adaptation approaches.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale
A persistent challenge in sign language video processing, including the task of sign language to written language translation, is how we learn representations of sign language in an effective and efficient way that can preserve the important attributes of these languages, while remaining invariant to irrelevant visual differences.
Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing
We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers.
Generative Context-aware Fine-tuning of Self-supervised Speech Models
Considering the recent advances in generative large language models (LLM), we hypothesize that an LLM could generate useful context information using the preceding text.
Toward Joint Language Modeling for Speech Units and Text
However, in the field of language modeling, very little effort has been made to model them jointly.
Audio-Visual Neural Syntax Acquisition
We study phrase structure induction from visually-grounded speech.
Few-Shot Spoken Language Understanding via Joint Speech-Text Models
Recent work on speech representation models jointly pre-trained with text has demonstrated the potential of improving speech representations by encoding speech and text in a shared space.
UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions
Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model's behavior and surpassing performance of task-specific models.
Ranked #1 on
Spoken Language Understanding
on Fluent Speech Commands
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement
In this work, we introduce AV2Wav, a resynthesis-based audio-visual speech enhancement approach that can generate clean speech despite the challenges of real-world training data.
Self-Supervised Video Transformers for Isolated Sign Language Recognition
This paper presents an in-depth analysis of various self-supervision methods for isolated sign language recognition (ISLR).
What Do Self-Supervised Speech Models Know About Words?
Many self-supervised speech models (S3Ms) have been introduced over the last few years, improving performance and data efficiency on various speech tasks.
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks
In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape.
Context-aware Fine-tuning of Self-supervised Speech Models
During the fine-tuning stage, we introduce an auxiliary loss that encourages this context embedding vector to be similar to context vectors of surrounding segments.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Comparative layer-wise analysis of self-supervised speech models
We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Open-Domain Sign Language Translation Learned from Online Video
Existing work on sign language translation - that is, translation from sign language videos into sentences in a written language - has focused mainly on (1) data collected in a controlled environment or (2) data in a specific domain, which limits the applicability to real-world settings.
Ranked #2 on
Gloss-free Sign Language Translation
on OpenASL
Gloss-free Sign Language Translation
Sign Language Translation
+1
Self-Supervised Speech Representation Learning: A Review
Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Searching for fingerspelled content in American Sign Language
This is an important task since significant content in sign language is often conveyed via fingerspelling, and to our knowledge the task has not been studied before.
On the Use of External Data for Spoken Named Entity Recognition
In this work we focus on low-resource spoken named entity recognition (NER) and address the question: Beyond self-supervised pre-training, how can we use external speech and/or text data that are not annotated for the task?
SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech
Historically these have focused on automatic speech recognition (ASR), speaker identification, or other lower-level tasks.
Ranked #1 on
Named Entity Recognition (NER)
on SLUE
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+7
Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing
We present substructure distribution projection (SubDP), a technique that projects a distribution over structures in one domain to another, by projecting substructure distributions separately.
On Generalization in Coreference Resolution
While coreference resolution is defined independently of dataset domain, most models for performing coreference resolution do not transfer well to unseen domains.
Ranked #1 on
Coreference Resolution
on Quizbowl
Layer-wise Analysis of a Self-supervised Speech Representation Model
Recently proposed self-supervised learning approaches have been successful for pre-training speech representation models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Fingerspelling Detection in American Sign Language
We propose a benchmark and a suite of evaluation metrics, some of which reflect the effect of detection on the downstream fingerspelling recognition task.
Chess as a Testbed for Language Model State Tracking
Motivated by this issue, we consider the task of language modeling for the game of chess.
Substructure Substitution: Structured Data Augmentation for NLP
We study a family of data augmentation methods, substructure substitution (SUB2), for natural language processing (NLP) tasks.
Learning Chess Blindfolded
Motivated by this issue, we consider the task of language modeling for the game of chess.
A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings
We propose a new unsupervised model for mapping a variable-duration speech segment to a fixed-dimensional representation.
Acoustic span embeddings for multilingual query-by-example search
In this work, we generalize AWE training to spans of words, producing acoustic span embeddings (ASE), and explore the application of ASE to QbE with arbitrary-length queries in multiple unseen languages.
Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks
Long document coreference resolution remains a challenging task due to the large memory and runtime requirements of current models.
Ranked #10 on
Coreference Resolution
on CoNLL 2012
On the Role of Supervision in Unsupervised Constituency Parsing
We analyze several recent unsupervised constituency parsing models, which are tuned with respect to the parsing $F_1$ score on the Wall Street Journal (WSJ) development set (1, 700 sentences).
Whole-Word Segmental Speech Recognition with Acoustic Word Embeddings
We find that word error rate can be reduced by a large margin by pre-training the acoustic segment representation with AWEs, and additional (smaller) gains can be obtained by pre-training the word prediction layer with AGWEs.
Multilingual Jointly Trained Acoustic and Written Word Embeddings
The pre-trained models can then be used for unseen zero-resource languages, or fine-tuned on data from low-resource languages.
Discrete Latent Variable Representations for Low-Resource Text Classification
While much work on deep latent variable models of text uses continuous latent variables, discrete latent variables are interesting because they are more interpretable and typically more space efficient.
A Cross-Task Analysis of Text Span Representations
Many natural language processing (NLP) tasks involve reasoning with textual spans, including question answering, entity recognition, and coreference resolution.
PeTra: A Sparsely Supervised Memory Model for People Tracking
We propose PeTra, a memory-augmented neural network designed to track entities in its memory slots.
Ranked #1 on
Coreference Resolution
on GAP
(F1 metric)
Fingerspelling recognition in the wild with iterative visual attention
In this paper we focus on recognition of fingerspelling sequences in American Sign Language (ASL) videos collected in the wild, mainly from YouTube and Deaf social media.
Variational Sequential Labelers for Semi-Supervised Learning
Our model family consists of a latent-variable generative model and a discriminative labeler.
Ranked #73 on
Named Entity Recognition (NER)
on CoNLL 2003 (English)
Visually Grounded Neural Syntax Acquisition
We define concreteness of constituents by their matching scores with images, and use it to guide the parsing of text.
Variational recurrent models for representation learning
Existing variational recurrent models typically use stochastic recurrent connections to model the dependence among neighboring latent variables, while generation assumes independence of generated data per time step given the latent sequence.
On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrieval
Recent work has shown that speech paired with images can be used to learn semantically meaningful speech representations even without any textual supervision.
Semantic query-by-example speech search using visual grounding
A number of recent studies have started to investigate how speech systems can be trained on untranscribed speech by leveraging accompanying images at training time.
Acoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognition
Direct acoustics-to-word (A2W) systems for end-to-end automatic speech recognition are simpler to train, and more efficient to decode with, than sub-word systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
American Sign Language fingerspelling recognition in the wild
As the first attempt at fingerspelling recognition in the wild, this work is intended to serve as a baseline for future work on sign language recognition in realistic conditions.
Pre-training on high-resource speech recognition improves low-resource speech-to-text translation
Finally, we show that the approach improves performance on a true low-resource task: pre-training on a combination of English ASR and French ASR improves Mboshi-French ST, where only 4 hours of data are available, from 3. 5 to 7. 1
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition
In this paper, we compare a suite of past methods and some of our own proposed methods for using unpaired text data to improve encoder-decoder models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Hierarchical Multitask Learning for CTC-based Speech Recognition
Previous work has shown that neural encoder-decoder speech recognition can be improved with hierarchical multitask learning, where auxiliary tasks are added at intermediate layers of a deep encoder.
Low-Resource Speech-to-Text Translation
We explore models trained on between 20 and 160 hours of data, and find that although models trained on less data have considerably lower BLEU scores, they can still predict words with relatively high precision and recall---around 50% for a model trained on 50 hours of data, versus around 60% for the full 160 hour model.
Acoustic feature learning using cross-domain articulatory measurements
Previous work has shown that it is possible to improve speech recognition by learning acoustic features from paired acoustic-articulatory data, for example by using canonical correlation analysis (CCA) or its deep extensions.
A Study of All-Convolutional Encoders for Connectionist Temporal Classification
We explore whether deep convolutional neural networks (CNNs) can be used effectively instead of RNNs as the "encoder" in CTC.
Multitask training with unlabeled data for end-to-end sign language fingerspelling recognition
We introduce a model for fingerspelling recognition that addresses these issues.
Semantic speech retrieval with a visually grounded model of untranscribed speech
We introduce a newly collected data set of human semantic relevance judgements and an associated task, semantic speech retrieval, where the goal is to search for spoken utterances that are semantically relevant to a given text query.
Acoustic Feature Learning via Deep Variational Canonical Correlation Analysis
We study the problem of acoustic feature learning in the setting where we have access to another (non-acoustic) modality for feature learning but not at test time.
End-to-End Neural Segmental Models for Speech Recognition
Segmental models are an alternative to frame-based models for sequence prediction, where hypothesized path weights are based on entire segment scores rather than a single frame at a time.
Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings
Query-by-example search often uses dynamic time warping (DTW) for comparing queries and proposed matching segments.
Learning to Embed Words in Context for Syntactic Tasks
We present models for embedding words in the context of surrounding words.
Parsing Speech: A Neural Approach to Integrating Lexical and Acoustic-Prosodic Information
In conversational speech, the acoustic signal provides cues that help listeners disambiguate difficult parses.
Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition
We hypothesize that using intermediate representations as auxiliary supervision at lower levels of deep networks may be a good way of combining the advantages of end-to-end training and more traditional pipeline approaches.
Visually grounded learning of keyword prediction from untranscribed speech
In this setting of images paired with untranscribed spoken captions, we consider whether computer vision systems can be used to obtain textual labels for the speech.
An embedded segmental K-means model for unsupervised segmentation and clustering of speech
Unsupervised segmentation and clustering of unlabelled speech are core problems in zero-resource speech processing.
Multi-view Recurrent Neural Acoustic Word Embeddings
Recent work has begun exploring neural acoustic word embeddings---fixed-dimensional vector representations of arbitrary-length speech segments corresponding to words.
Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches
Acoustic word embeddings --- fixed-dimensional vector representations of variable-length spoken word segments --- have begun to be considered for tasks such as speech recognition and query-by-example search.
End-to-End Training Approaches for Discriminative Segmental Models
Similarly to hybrid HMM-neural network models, segmental models of this class can be trained in two stages (frame classifier training followed by linear segmental model weight training), end to end (joint training of both frame classifier and linear weights), or with end-to-end fine-tuning after two-stage training.
Jointly Learning to Align and Convert Graphemes to Phonemes with Neural Attention Models
We propose an attention-enabled encoder-decoder model for the problem of grapheme-to-phoneme conversion.
Deep Variational Canonical Correlation Analysis
We present deep variational canonical correlation analysis (VCCA), a deep multi-view learning model that extends the latent variable model interpretation of linear CCA to nonlinear observation models parameterized by deep neural networks.
Lexicon-Free Fingerspelling Recognition from Video: Data, Models, and Signer Adaptation
Recognizing fingerspelling is challenging for a number of reasons: It involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected.
Efficient Segmental Cascades for Speech Recognition
Discriminative segmental models offer a way to incorporate flexible feature functions into speech recognition.
Charagram: Embedding Words and Sentences via Character n-grams
We present Charagram embeddings, a simple approach for learning character-based compositional models to embed textual sequences.
Signer-independent Fingerspelling Recognition with Deep Neural Network Adaptation
Previous work has shown that it is possible to achieve almost 90% accuracies on fingerspelling recognition in a signer-dependent setting.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
On Deep Multi-View Representation Learning: Objectives and Optimization
We consider learning representations (features) in the setting in which we have access to multiple unlabeled views of the data for learning while only one view is available for downstream tasks.
Towards Universal Paraphrastic Sentence Embeddings
We again find that the word averaging models perform well for sentence similarity and entailment, outperforming LSTMs.
Nonparametric Canonical Correlation Analysis
Several nonlinear extensions of the original linear CCA have been proposed, including kernel and deep neural network methods.
Large-Scale Approximate Kernel Canonical Correlation Analysis
Kernel canonical correlation analysis (KCCA) is a nonlinear multi-view representation learning technique with broad applicability in statistics and machine learning.
Mapping Unseen Words to Task-Trained Embedding Spaces
We consider the supervised training setting in which we learn task-specific word embeddings.
Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations
Deep CCA is a recently proposed deep neural network extension to the traditional canonical correlation analysis (CCA), and has been successful for multi-view representation learning in several domains.
Deep convolutional acoustic word embeddings using word-pair side information
Recent studies have been revisiting whole words as the basic modelling unit in speech recognition and query applications, instead of phonetic units.
Discriminative Segmental Cascades for Feature-Rich Phone Recognition
A typical solution is to use approximate decoding, either by beam pruning in a single pass or by beam pruning to generate a lattice followed by a second pass.
From Paraphrase Database to Compositional Paraphrase Model and Back
The Paraphrase Database (PPDB; Ganitkevitch et al., 2013) is an extensive semantic resource, consisting of a list of phrase pairs with (heuristic) confidence estimates.
