Search Results for author:
Found 83 papers, 25 papers with code
Zero-shot Learning for Grapheme to Phoneme Conversion with Language Ensemble
Grapheme-to-Phoneme (G2P) has many applications in NLP and speech fields.
Phone Inventories and Recognition for Every Language
Identifying phone inventories is a crucial component in language documentation and the preservation of endangered languages.
CMU’s Machine Translation System for IWSLT 2019
In Neural Machine Translation (NMT) the usage of sub-words and characters as source and target units offers a simple and flexible solution for translation of rare and unseen words.
Error-aware Quantization through Noise Tempering
In this work, we incorporate exponentially decaying quantization-error-aware noise together with a learnable scale of task loss gradient to approximate the effect of a quantization operator.
Normalized Contrastive Learning for Text-Video Retrieval
Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness.
Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models
End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation.
SQuAT: Sharpness- and Quantization-Aware Training for BERT
Quantization is an effective technique to reduce memory footprint, inference latency, and power consumption of deep learning models.
CTC Alignments Improve Autoregressive Translation
Connectionist Temporal Classification (CTC) is a widely used approach for automatic speech recognition (ASR) that performs conditionally independent monotonic alignment.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
ASR2K: Speech Recognition for Around 2000 Languages without Audio
We achieve 50% CER and 74% WER on the Wilderness dataset with Crubadan statistics only and improve them to 45% CER and 69% WER when using 10000 raw text utterances.
Masked Autoencoders that Listen
Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers.
Ranked #2 on
Speaker Identification
on VoxCeleb1
LegoNN: Building Modular Encoder-Decoder Models
We present several experiments to demonstrate the effectiveness of LegoNN models: a trained language generation LegoNN decoder module from German-English (De-En) MT task can be reused with no fine-tuning for the Europarl English ASR and the Romanian-English (Ro-En) MT tasks to match or beat respective baseline models.
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
Combining the visual modality with pretrained language models has been surprisingly effective for simple descriptive tasks such as image captioning.
Speech Summarization using Restricted Self-Attention
End-to-end modeling of speech summarization models is challenging due to memory and compute constraints arising from long input audio sequences.
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks.
Ranked #1 on
Zero-Shot Video Retrieval
on YouCook2
Differentiable Allophone Graphs for Language-Universal Speech Recognition
These phone-based systems with learned allophone graphs can be used by linguists to document new languages, build phone-based lexicons that capture rich pronunciation variations, and re-evaluate the allophone mappings of seen language.
Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding
Our splits identify performance gaps up to 10% between end-to-end systems that were within 1% of each other on the original test sets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
In video transformers, the time dimension is often treated in the same way as the two spatial dimensions.
Ranked #10 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks.
Ranked #2 on
Temporal Action Localization
on CrossTask
(using extra training data)
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
In this work, we present an end-to-end framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks.
Self-supervised object detection from audio-visual correspondence
We tackle the problem of learning object detectors without supervision.
Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning
First, for space, we show that spatial augmentations such as cropping do work well for videos too, but that previous implementations, due to the high processing and memory cost, could not do this at a scale sufficient for it to work well.
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextualized multilingual multimodal embeddings.
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering
When Question-Answering (QA) systems are deployed in the real world, users query them through a variety of interfaces, such as speaking to voice assistants, typing questions into a search engine, or even translating questions to languages supported by the QA system.
End-to-end Quantized Training via Log-Barrier Extensions
Quantization of neural network parameters and activations has emerged as a successful approach to reducing the model size and inference time on hardware that sup-ports native low-precision arithmetic.
Audio-Visual Event Recognition through the lens of Adversary
This work aims to study several key questions related to multimodal learning through the lens of adversarial noises: 1) The trade-off between early/middle/late fusion affecting its robustness and accuracy 2) How do different frequency/time domain features contribute to the robustness?
Multimodal Speech Recognition with Unstructured Audio Masking
Our experiments on the Flickr 8K Audio Captions Corpus show that multimodal ASR can generalize to recover different types of masked words in this unstructured masking setting.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
On Long-Tailed Phenomena in Neural Machine Translation
State-of-the-art Neural Machine Translation (NMT) models struggle with generating low-frequency tokens, tackling which remains a major challenge.
Support-set bottlenecks for video-text representation learning
The dominant paradigm for learning video-text representations -- noise contrastive learning -- increases the similarity of the representations of pairs of samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs.
Fine-Grained Grounding for Multimodal Speech Recognition
In experiments on the Flickr8K Audio Captions Corpus, we find that our model improves over approaches that use global visual features, that the proposals enable the model to recover entities and other related words, such as adjectives, and that improvements are due to the model's ability to localize the correct proposals.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Revisiting Factorizing Aggregated Posterior in Learning Disentangled Representations
In this paper, we provide a theoretical explanation that low total correlation of sampled representation cannot guarantee low total correlation of the mean representation.
How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language
Towards this end, we introduce How2Sign, a multimodal and multiview continuous American Sign Language (ASL) dataset, consisting of a parallel corpus of more than 80 hours of sign language videos and a set of corresponding modalities including speech, English transcripts, and depth.
Contextual RNN-T For Open Domain ASR
By using an attention model and a biasing model to leverage the contextual metadata that accompanies a video, we observe a relative improvement of about 16% in Word Error Rate on Named Entities (WER-NE) for videos with related metadata.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
AlloVera: A Multilingual Allophone Database
While phonemic representations are language specific, phonetic representations (stated in terms of (allo)phones) are much closer to a universal (language-independent) transcription.
ASR Error Correction and Domain Adaptation Using Machine Translation
We propose a simple technique to perform domain adaptation for ASR error correction via machine translation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Universal Phone Recognition with a Multilingual Allophone System
Multilingual models can improve language processing, particularly for low resource situations, by sharing parameters across languages.
Towards Zero-shot Learning for Automatic Phonemic Transcription
The difficulty of this task is that phoneme inventories often differ between the training languages and the target language, making it infeasible to recognize unseen phonemes.
Looking Enhances Listening: Recovering Missing Speech Using Images
Speech is understood better by using visual context; for this reason, there have been many attempts to use images to adapt automatic speech recognition (ASR) systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Gun Source and Muzzle Head Detection
We have interesting results both in bounding the shooter as well as detecting the gun smoke.
Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
Inspired by modular software design principles of independence, interchangeability, and clarity of interface, we introduce a method for enforcing encoder-decoder modularity in seq2seq models without sacrificing the overall model quality or its full differentiability.
On Compositionality in Neural Machine Translation
We investigate two specific manifestations of compositionality in Neural Machine Translation (NMT) : (1) Productivity - the ability of the model to extend its predictions beyond the observed length in training data and (2) Systematicity - the ability of the model to systematically recombine known parts and rules.
Adversarial Music: Real World Audio Adversary Against Wake-word Detection System
In this work, we target our attack on the wake-word detection system, jamming the model with some inconspicuous background music to deactivate the VAs while our audio adversary is present.
Multitask Learning For Different Subword Segmentations In Neural Machine Translation
In Neural Machine Translation (NMT) the usage of subwords and characters as source and target units offers a simple and flexible solution for translation of rare and unseen words.
On Leveraging the Visual Modality for Neural Machine Translation
Leveraging the visual modality effectively for Neural Machine Translation (NMT) remains an open problem in computational linguistics.
On Dimensional Linguistic Properties of the Word Embedding Space
Word embeddings have become a staple of several natural language processing tasks, yet much remains to be understood about their properties.
RTC-VAE: HARNESSING THE PECULIARITY OF TOTAL CORRELATION IN LEARNING DISENTANGLED REPRESENTATIONS
In the problem of unsupervised learning of disentangled representations, one of the promising methods is to penalize the total correlation of sampled latent vari-ables.
SANTLR: Speech Annotation Toolkit for Low Resource Languages
In this work, we present SANTLR: Speech Annotation Toolkit for Low Resource Languages.
Multilingual Speech Recognition with Corpus Relatedness Sampling
For example, the target corpus might benefit more from a corpus in the same domain or a corpus from a close language.
Effective Dimensionality Reduction for Word Embeddings
Pre-trained word embeddings are used in several downstream applications as well as for constructing representations for sentences, paragraphs and documents.
Cross-Attention End-to-End ASR for Two-Party Conversations
We present an end-to-end speech recognition model that learns interaction between two speakers based on the turn-changing information.
Analyzing Utility of Visual Context in Multimodal Speech Recognition Under Noisy Conditions
Multimodal learning allows us to leverage information from multiple sources (visual, acoustic and text), similar to our experience of the real world.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion
We present a novel conversational-context aware end-to-end speech recognizer based on a gated neural network that incorporates conversational-context/word/speech embeddings.
Multimodal Abstractive Summarization for How2 Videos
In this paper, we study abstractive summarization for open-domain videos.
Ranked #1 on
Text Summarization
on How2
Acoustic-to-Word Models with Conversational Context Information
Conversational context information, higher-level knowledge that spans across sentences, can help to recognize a long conversation.
The ARIEL-CMU Systems for LoReHLT18
This paper describes the ARIEL-CMU submissions to the Low Resource Human Language Technologies (LoReHLT) 2018 evaluations for the tasks Machine Translation (MT), Entity Discovery and Linking (EDL), and detection of Situation Frames in Text and Speech (SF Text and Speech).
Phoneme Level Language Models for Sequence Based Low Resource ASR
Building multilingual and crosslingual models help bring different languages together in a language universal space.
Learned In Speech Recognition: Contextual Acoustic Word Embeddings
End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not require a lexicon.
Learning from Multiview Correlations in Open-Domain Videos
This shows it is possible to learn reliable representations across disparate, unaligned and noisy modalities, and encourages using the proposed approach on larger datasets.
Multimodal Grounding for Sequence-to-Sequence Speech Recognition
Specifically, in our previous work, we propose a multistep visual adaptive training approach which improves the accuracy of an audio-based Automatic Speech Recognition (ASR) system.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
How2: A Large-scale Dataset for Multimodal Language Understanding
In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Connectionist Temporal Localization for Sound Event Detection with Sequential Labeling
Research on sound event detection (SED) with weak labeling has mostly focused on presence/absence labeling, which provides no temporal information at all about the event occurrences.
Sound Audio and Speech Processing
A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling
This paper compares five types of pooling functions both theoretically and experimentally, with special focus on their performance of localization.
Sound Audio and Speech Processing
Zero-shot Learning for Speech Recognition with Universal Phonetic Model
Our model is able to recognize unseen phonemes in the target language, if only a small text corpus is available.
Activity Recognition on a Large Scale in Short Videos - Moments in Time Dataset
Moments capture a huge part of our lives.
Dialog-context aware end-to-end speech recognition
Existing speech recognition systems are typically built at the sentence level, although it is known that dialog context, e. g. higher-level knowledge that spans across sentences or speakers, can help the processing of long conversations.
Domain Robust Feature Extraction for Rapid Low Resource ASR Development
We demonstrate the effectiveness of using a pre-trained English recognizer, which is robust to such mismatched conditions, as a domain normalizing feature extractor on a low resource language.
Acoustic-to-Word Recognition with Sequence-to-Sequence Models
We present effective methods to train Sequence-to-Sequence models for direct word-level recognition (and character-level recognition) and show an absolute improvement of 4. 4-5. 0\% in Word Error Rate on the Switchboard corpus compared to prior work.
Hierarchical Multi Task Learning With CTC
Our model obtains 14. 0% Word Error Rate on the Eval2000 Switchboard subset without any decoder or language model, outperforming the current state-of-the-art on acoustic-to-word models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval
Constructing a joint representation invariant across different modalities (e. g., video, language) is of significant importance in many multimedia applications.
Ranked #28 on
Video Retrieval
on MSR-VTT
End-to-End Multimodal Speech Recognition
Transcription or sub-titling of open-domain videos is still a challenging domain for Automatic Speech Recognition (ASR) due to the data's challenging acoustics, variable signal processing and the essentially unrestricted domain of the data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Sequence-based Multi-lingual Low Resource Speech Recognition
Techniques for multi-lingual and cross-lingual speech recognition can help in low resource scenarios, to bootstrap systems and enable analysis of new languages and domains.
Linguistic unit discovery from multi-modal inputs in unwritten languages: Summary of the "Speaking Rosetta" JSALT 2017 Workshop
We summarize the accomplishments of a multi-disciplinary workshop exploring the computational and scientific issues surrounding the discovery of linguistic units (subwords and words) in a language without orthography.
Subword and Crossword Units for CTC Acoustic Models
This paper proposes a novel approach to create an unit set for CTC based speech recognition systems.
Visual Features for Context-Aware Speech Recognition
We are working on a corpus of "how-to" videos from the web, and the idea is that an object that can be seen ("car"), or a scene that is being detected ("kitchen") can be used to condition both models on the "context" of the recording, thereby reducing perplexity and improving transcription.
Annotating High-Level Structures of Short Stories and Personal Anecdotes
Stories are a vital form of communication in human culture; they are employed daily to persuade, to elicit sympathy, or to convey a message.
Cultural Vocal Bursts Intensity Prediction
Vocal Bursts Intensity Prediction
Comparison of Decoding Strategies for CTC Acoustic Models
The CTC loss function maps an input sequence of observable feature vectors to an output sequence of symbols.
A Comparison of deep learning methods for environmental sound
On these features, we apply five models: Gaussian Mixture Model (GMM), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Convolutional Deep Neural Net- work (CNN) and i-vector.
Robust end-to-end deep audiovisual speech recognition
Speech is one of the most effective ways of communication among humans.
EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
The performance of automatic speech recognition (ASR) has improved tremendously due to the application of deep neural networks (DNNs).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
