Search Results for author:
Found 76 papers, 46 papers with code
DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts
The proliferation of disinformation presents a growing threat to societal trust and democracy, necessitating robust and scalable Fact-Checking systems.
InFact: A Strong Baseline for Automated Fact-Checking
The spread of disinformation poses a global threat to democratic societies, necessitating robust and scalable Automated Fact-Checking (AFC) systems.
Ranked #3 on
Fact Checking
on AVeriTeC
The Surprising Effectiveness of Multimodal Large Language Models for Video Moment Retrieval
We achieve a new state-of-the-art in moment retrieval on the widely used benchmarks Charades-STA, QVHighlights, and ActivityNet Captions.
Efficient Pre-training for Localized Instruction Generation of Videos
To mitigate these issues, we propose a novel technique, Sieve-&-Swap, to automatically generate high-quality training data for the recipe domain: (i) Sieve: filters irrelevant transcripts and (ii) Swap: acquires high-quality text by replacing transcripts with human-written instruction from a text-only recipe dataset.
Improving Selective Visual Question Answering by Learning from Your Peers
In this work, we explore Selective VQA in both in-distribution (ID) and OOD scenarios, where models are presented with mixtures of ID and OOD data.
Simple Token-Level Confidence Improves Caption Correctness
The ability to judge whether a caption correctly describes an image is a critical part of vision-language understanding.
Ranked #62 on
Visual Reasoning
on Winoground
Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition
We propose to learn what makes a good video for action recognition and select only high-quality samples for augmentation.
Ranked #2 on
Few Shot Action Recognition
on HMDB51
Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly
We first enable abstention capabilities for several VQA models, and analyze both their coverage, the portion of questions answered, and risk, the error on that portion.
Learning To Recognize Procedural Activities with Distant Supervision
In this paper we consider the problem of classifying fine-grained, multi-step activities (e. g., cooking different recipes, making disparate home improvements, creating various forms of arts and crafts) from long videos spanning up to several minutes.
Ranked #4 on
Video Classification
on COIN
FLAVA: A Foundational Language And Vision Alignment Model
State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks.
Ranked #4 on
Image Retrieval
on MS COCO
A New Split for Evaluating True Zero-Shot Action Recognition
We benchmark several recent approaches on the proposed True Zero-Shot(TruZe) Split for UCF101 and HMDB51, with zero-shot and generalized zero-shot evaluation.
CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition
Theproblem can be seen as learning a function which general-izes well to instances of unseen classes without losing dis-crimination between classes.
Ranked #2 on
Zero-Shot Action Recognition
on Olympics
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image.
Ranked #8 on
Visual Question Answering (VQA)
on A-OKVQA
SMART Frame Selection for Action Recognition
In this work, however, we focus on the more standard short, trimmed action recognition problem.
Ranked #6 on
Action Recognition
on UCF101
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting
The goal of continual learning (CL) is to learn a sequence of tasks without suffering from the phenomenon of catastrophic forgetting.
TextCaps: a Dataset for Image Captioning with Reading Comprehension
Image descriptions can help visually impaired people to quickly understand the image content.
Adversarial Continual Learning
We show that shared features are significantly less prone to forgetting and propose a novel hybrid continual learning framework that learns a disjoint representation for task-invariant and task-specific features required to solve a sequence of tasks.
In Defense of Grid Features for Visual Question Answering
Popularized as 'bottom-up' attention, bounding box (or region) based visual features have recently surpassed vanilla grid-based convolutional features as the de facto standard for vision and language tasks like visual question answering (VQA).
Ranked #18 on
Visual Question Answering (VQA)
on VQA v2 test-std
12-in-1: Multi-Task Vision and Language Representation Learning
Much of vision-and-language research focuses on a small but diverse set of independent tasks and supporting datasets often studied in isolation; however, the visually-grounded language understanding skills required for success at these tasks overlap significantly.
Ranked #1 on
10-shot image generation
on MEAD
Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question.
Decoupling Representation and Classifier for Long-Tailed Recognition
The long-tail distribution of the visual world poses great challenges for deep learning based classification models on how to handle the class imbalance problem.
Ranked #3 on
Long-tail learning with class descriptors
on CUB-LT
Uncertainty-guided Continual Learning with Bayesian Neural Networks
Continual learning aims to learn new tasks without forgetting previously learned ones.
Learning to Generate Grounded Visual Captions without Localization Supervision
When automatically generating a sentence description for an image or video, it often remains unclear how well the generated caption is grounded, that is whether the model uses the correct image regions to output particular words, or if the model is hallucinating based on priors in the dataset and/or the language model.
Towards VQA Models That Can Read
We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset.
Ranked #3 on
Visual Question Answering (VQA)
on VizWiz 2018
Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
Similarly, the output feature maps of a convolution layer can also be seen as a mixture of information at different frequencies.
Ranked #149 on
Action Classification
on Kinetics-400
CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
Specifically, we construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset.
On Tiny Episodic Memories in Continual Learning
But for a successful knowledge transfer, the learner needs to remember how to perform previous tasks.
Ranked #7 on
Class Incremental Learning
on cifar100
Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering
We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable.
Cycle-Consistency for Robust Visual Question Answering
Despite significant progress in Visual Question Answering over the years, robustness of today's VQA models leave much to be desired.
DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition
Motion has shown to be useful for video understanding, where motion is typically represented by optical flow.
Ranked #1 on
Action Recognition
on UCF-101
Exploring the Challenges towards Lifelong Fact Learning
So far life-long learning (LLL) has been studied in relatively small-scale and relatively artificial setups.
Grounded Video Description
Our dataset, ActivityNet-Entities, augments the challenging ActivityNet Captions dataset with 158k bounding box annotations, each grounding a noun phrase.
Adversarial Inference for Multi-Sentence Video Description
Among the main issues are the fluency and coherence of the generated descriptions, and their relevance to the video.
Efficient Lifelong Learning with A-GEM
In lifelong learning, the learner is presented with a sequence of tasks, incrementally building a data-driven prior which may be leveraged to speed up learning of a new task.
Ranked #6 on
Continual Learning
on ASC (19 tasks)
Graph-Based Global Reasoning Networks
In this work, we propose a new approach for reasoning globally in which a set of features are globally aggregated over the coordinate space and then projected to an interaction space where relational reasoning can be efficiently computed.
A Dataset for Telling the Stories of Social Media Videos
Video content on social media platforms constitutes a major part of the communication between people, as it allows everyone to share their stories.
Uncertainty-guided Lifelong Learning in Bayesian Networks
Sequentially learning of tasks arriving in a continuous stream is a complex problem and becomes more challenging when the model has a fixed capacity.
Visual Coreference Resolution in Visual Dialog using Neural Module Networks
Visual dialog entails answering a series of questions grounded in an image, using dialog history as context.
Ranked #1 on
Common Sense Reasoning
on Visual Dialog v0.9
Pythia v0.1: the Winning Entry to the VQA Challenge 2018
We demonstrate that by making subtle but important changes to the model architecture and the learning rate schedule, fine-tuning image features, and adding data augmentation, we can significantly improve the performance of the up-down model on VQA v2. 0 dataset -- from 65. 67% to 70. 22%.
Ranked #12 on
Visual Question Answering (VQA)
on A-OKVQA
Selfless Sequential Learning
In particular, we propose a novel regularizer, that encourages representation sparsity by means of neural inhibition.
Large-Scale Visual Relationship Understanding
Large scale visual understanding is challenging, as it requires a model to handle the widely-spread and imbalanced distribution of <subject, relation, object> triples.
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
We propose a multimodal approach to explanation, and argue that the two modalities provide complementary explanatory strengths.
CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
The game involves two players: a Teller and a Drawer.
Memory Aware Synapses: Learning what (not) to forget
We show state-of-the-art performance and, for the first time, the ability to adapt the importance of the parameters based on unlabeled data towards what the network needs (not) to forget, which may vary depending on test conditions.
Attentive Explanations: Justifying Decisions and Pointing to the Evidence (Extended Abstract)
We also introduce a multimodal methodology for generating visual and textual explanations simultaneously.
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems.
Ranked #44 on
Visual Question Answering (VQA)
on VQA v2 test-dev
Generating Descriptions with Grounded and Co-Referenced People
At training time, we first learn how to localize characters by relating their visual appearance to mentions in the descriptions via a semi-supervised approach.
Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
While strong progress has been made in image captioning over the last years, machine and human captions are still quite distinct.
Attentive Explanations: Justifying Decisions and Pointing to the Evidence
In contrast, humans can justify their decisions with natural language and point to the evidence in the visual world which led to their decisions.
Modeling Relationships in Referential Expressions with Compositional Modular Networks
In this paper we instead present a modular deep architecture capable of analyzing referential expressions into their component parts, identifying entities and relationships mentioned in the input expression and grounding them all in the scene.
Ranked #1 on
Visual Question Answering (VQA)
on Visual7W
Utilizing Large Scale Vision and Text Datasets for Image Segmentation from Referring Expressions
Image segmentation from referring expressions is a joint vision and language modeling task, where the input is an image and a textual expression describing a particular region in the image; and the goal is to localize and segment the specific image region based on the given expression.
Captioning Images with Diverse Objects
We propose minimizing a joint objective which can learn from these diverse data sources and leverage distributional semantic embeddings, enabling the model to generalize and describe novel objects outside of image-caption datasets.
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.
Movie Description
In addition we also collected and aligned movie scripts used in prior work and compare the two sources of descriptions.
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering
By combining latest advances in image representation and natural language processing, we propose Ask Your Neurons, a scalable, jointly trained, end-to-end formulation to this problem.
Attributes as Semantic Units between Natural Language and Visual Recognition
Impressive progress has been made in the fields of computer vision and natural language processing.
Generating Visual Explanations
Clearly explaining a rationale for a classification decision to an end-user can be as important as the decision itself.
Segmentation from Natural Language Expressions
To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information.
Ranked #16 on
Referring Expression Segmentation
on J-HMDB
Learning to Compose Neural Networks for Question Answering
We describe a question answering model that applies to both images and structured knowledge bases.
Sequence to Sequence - Video to Text
Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip.
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet.
Natural Language Object Retrieval
In this paper, we address the task of natural language object retrieval, to localize a target object within a given image based on a natural language query of the object.
Ranked #12 on
Referring Expression Comprehension
on Talk2Car
Grounding of Textual Phrases in Images by Reconstruction
We propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly.
Ranked #13 on
Phrase Grounding
on Flickr30k Entities Test
Neural Module Networks
Visual question answering is fundamentally compositional in nature---a question like "where is the dog?"
Ranked #6 on
Visual Question Answering (VQA)
on VQA v1 test-std
Spatial Semantic Regularisation for Large Scale Object Detection
Our approach proves to be especially useful in large scale settings with thousands of classes, where spatial and semantic interactions are very frequent and only weakly supervised detectors can be built due to a lack of bounding box annotations.
The Long-Short Story of Movie Description
Generating descriptions for videos has many applications including assisting blind people and human-robot interaction.
A Multi-scale Multiple Instance Video Description Network
Most state-of-the-art methods for solving this problem borrow existing deep convolutional neural network (CNN) architectures (AlexNet, GoogLeNet) to extract a visual representation of the input video.
Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
In contrast to previous efforts, we are facing a multi-modal problem where the language output (answer) is conditioned on visual and natural language input (image and question).
Sequence to Sequence -- Video to Text
Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip.
Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Data
To attack the second challenge, recognizing composite activities, we leverage the fact that these activities are compositional and that the essential components of the activities can be obtained from textual descriptions or scripts.
A Dataset for Movie Description
In this work we propose a novel dataset which contains transcribed DVS, which is temporally aligned to full length HD movies.
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Solving the visual symbol grounding problem has long been a goal of artificial intelligence.
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or "temporally deep", are effective for tasks involving sequences, visual and otherwise.
Ranked #3 on
Human Interaction Recognition
on BIT
Coherent Multi-Sentence Video Description with Variable Level of Detail
Humans can easily describe what they see in a coherent way and at varying level of detail.
Transfer Learning in a Transductive Setting
Our approach consistently outperforms state-of-the-art transfer and semi-supervised approaches on all datasets.
Grounding Action Descriptions in Videos
Recent work has shown that the integration of visual information into text-based models can substantially improve model predictions, but so far only visual information extracted from static images has been used.
