Search Results for author:
Found 17 papers, 7 papers with code
CMU’s Machine Translation System for IWSLT 2019
In Neural Machine Translation (NMT) the usage of sub-words and characters as source and target units offers a simple and flexible solution for translation of rare and unseen words.
Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
Prior work on selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain.
WinoViz: Probing Visual Properties of Objects Under Different States
We present WinoViz, a text-only evaluation dataset, consisting of 1, 380 examples that probe the reasoning abilities of language models regarding variant visual properties of objects under different contexts or states.
Exploring Strategies for Modeling Sign Language Phonology
Specifically, we explore how learning strategies like multi-task and curriculum learning can leverage mutually useful information between phoneme types to facilitate better modeling of sign language phonemes.
I2I: Initializing Adapters with Improvised Knowledge
We propose Improvise to Initialize (I2I), a continual learning algorithm that initializes Adapters for incoming tasks by distilling knowledge from previously-learned tasks' Adapters.
Multimodal Speech Recognition for Language-Guided Embodied Agents
Benchmarks for language-guided embodied agents typically assume text-based instructions, but deployed agents will encounter spoken instructions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
VAuLT: Augmenting the Vision-and-Language Transformer for Sentiment Classification on Social Media
VAuLT is an extension of the popular Vision-and-Language Transformer (ViLT), and improves performance on vision-and-language (VL) tasks that involve more complex text inputs than image captions while having minimal impact on training and inference efficiency.
Curriculum Learning for Data-Efficient Vision-Language Alignment
Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data.
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks
Existing CL benchmarks have facilitated research on task adaptation and mitigating "catastrophic forgetting", but are limited to vision-only and language-only tasks.
Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
Numerous works have analyzed biases in vision and pre-trained language models individually - however, less attention has been paid to how these biases interact in multimodal settings.
Reasoning Over History: Context Aware Visual Dialog
While neural models have been shown to exhibit strong performance on single-turn visual question answering (VQA) tasks, extending VQA to a multi-turn, conversational setting remains a challenge.
Multimodal Speech Recognition with Unstructured Audio Masking
Our experiments on the Flickr 8K Audio Captions Corpus show that multimodal ASR can generalize to recover different types of masked words in this unstructured masking setting.
Fine-Grained Grounding for Multimodal Speech Recognition
In experiments on the Flickr8K Audio Captions Corpus, we find that our model improves over approaches that use global visual features, that the proposals enable the model to recover entities and other related words, such as adjectives, and that improvements are due to the model's ability to localize the correct proposals.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Looking Enhances Listening: Recovering Missing Speech Using Images
Speech is understood better by using visual context; for this reason, there have been many attempts to use images to adapt automatic speech recognition (ASR) systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Multitask Learning For Different Subword Segmentations In Neural Machine Translation
In Neural Machine Translation (NMT) the usage of subwords and characters as source and target units offers a simple and flexible solution for translation of rare and unseen words.
Structured Fusion Networks for Dialog
Neural dialog models have exhibited strong performance, however their end-to-end nature lacks a representation of the explicit structure of dialog.
Analyzing Utility of Visual Context in Multimodal Speech Recognition Under Noisy Conditions
Multimodal learning allows us to leverage information from multiple sources (visual, acoustic and text), similar to our experience of the real world.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
