Search Results for author:
Found 104 papers, 57 papers with code
Object Manipulation via Visual Target Localization
Object manipulation is a critical skill required for Embodied AI agents interacting with the world around them.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin.
Ranked #1 on
Image Classification
on ImageNet V2
(using extra training data)
MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound
Given a video, we replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet.
Ranked #1 on
Action Classification
on Kinetics-600
(using extra training data)
The Introspective Agent: Interdependence of Strategy, Physiology, and Sensing for Embodied Agents
While traditional embodied agents manipulate an environment to best achieve a goal, we argue for an introspective agent, which considers its own abilities in the context of its environment.
Forward Compatible Training for Large-Scale Embedding Retrieval Systems
To avoid the cost of backfilling, BCT modifies training of the new model to make its representations compatible with those of the old model.
Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text
We investigate these challenges in the context of Iconary, a collaborative game of drawing and guessing based on Pictionary, that poses a novel challenge for the research community.
LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time
Our models require no retraining, thus our subspace of models can be deployed entirely on-device to allow adaptive network compression at inference time.
Robust fine-tuning of zero-shot models
Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution.
LanguageRefer: Spatial-Language Model for 3D Visual Grounding
Specifically, given a reconstructed 3D scene in the form of point clouds with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model successfully identifies the target object from a set of potential candidates.
MERLOT: Multimodal Neural Script Knowledge Models
As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future.
LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes
We further quantitatively measure the quality of our codes by applying it to the efficient image retrieval as well as out-of-distribution (OOD) detection problems.
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language.
Pushing it out of the Way: Interactive Visual Navigation
In this paper, we study the problem of interactive navigation where agents learn to change the environment to navigate more efficiently to their goals.
Learning Neural Network Subspaces
Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance.
Learning Visual Representation from Human Interactions
Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision.
Learning Flexible Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solving, decision making and socialization.
Layer-Wise Data-Free CNN Compression
In the high-computation regime, we show how to combine our method with generative methods to improve upon state-of-the-art performance for several networks.
What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision.
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks
Autonomous agents must learn to collaborate.
FLUID: A Unified Evaluation Framework for Flexible Sequential Data
To foster research towards the goal of general ML methods, we introduce a new unified evaluation framework - FLUID (Flexible Sequential Data).
Supermasks in Superposition
We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting.
Probing Contextual Language Models for Common Ground with Visual Representations
The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations.
VisualCOMET: Reasoning about the Dynamic Context of a Still Image
In addition, we provide person-grounding (i. e., co-reference links) between people appearing in the image and people mentioned in the textual commonsense descriptions, allowing for tighter integration between images and text.
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
We argue that interactive and embodied visual AI has reached a stage of development similar to visual recognition prior to the advent of these ecosystems.
TuringAdvice: A Generative and Dynamic Evaluation of Language Use
We propose TuringAdvice, a new challenge task and dataset for language understanding models.
Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects
Our quantitative and qualitative results show that (a) we can predict meaningful forces from videos whose effects lead to accurate imitation of the motions observed, (b) by jointly optimizing for contact point and force prediction, we can improve the performance on both tasks in comparison to independent training, and (c) we can learn a representation from this model that generalizes to novel objects using few shot examples.
Grounded Situation Recognition
We introduce Grounded Situation Recognition (GSR), a task that requires producing structured semantic summaries of images describing: the primary activity, entities engaged in the activity with their roles (e. g. agent, tool), and bounding-box groundings of entities.
Watching the World Go By: Representation Learning from Unlabeled Videos
Recent single image unsupervised representation learning techniques show remarkable success on a variety of tasks.
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
We publicly release all of our experimental data, including training and validation scores for 2, 100 trials, to encourage further analysis of training dynamics during fine-tuning.
Soft Threshold Weight Reparameterization for Learnable Sparsity
Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget.
Learning Generalizable Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solving, decision making, and socialization.
Visual Reaction: Learning to Play Catch with Your Drone
In this paper we address the problem of visual reaction: the task of interacting with dynamic environments where the changes in the environment are not necessarily caused by the agent itself.
What's Hidden in a Randomly Weighted Neural Network?
Training a neural network is synonymous with learning the values of the weights.
Ranked #547 on
Image Classification
on ImageNet
Conditional Driving from Natural Language Instructions
Widespread adoption of self-driving cars will depend not only on their safety but largely on their ability to interact with human users.
Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index
Existing open-domain question answering (QA) models are not suitable for real-time usage because they need to process several long documents on-demand for every input query.
Butterfly Transform: An Efficient FFT Based Neural Architecture Design
By replacing pointwise convolutions with BFT, we reduce the computational complexity of these layers from O(n^2) to O(n\log n) with respect to the number of channels.
Discovering Neural Wirings
In this work we propose a method for discovering neural wirings.
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources.
Defending Against Neural Fake News
We find that best current discriminators can classify neural fake news from real, human-written, news with 73% accuracy, assuming access to a moderate level of training data.
Ranked #2 on
Fake News Detection
on Grover-Mega
HellaSwag: Can a Machine Really Finish Your Sentence?
In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset.
Two Body Problem: Collaborative Visual Task Completion
Collaboration is a necessary skill to perform tasks that are beyond one agent's capabilities.
Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph
Visual relationship reasoning is a crucial yet challenging task for understanding rich interactions across visual concepts.
What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning
In this work we propose Hierarchical Planning and Reinforcement Learning (HIP-RL), a method for merging the benefits and capabilities of Symbolic Planning with the learning abilities of Deep Reinforcement Learning.
ELASTIC: Improving CNNs with Dynamic Scaling Policies
We formulate the scaling policy as a non-linear function inside the network's structure that (a) is learned from data, (b) is instance specific, (c) does not add extra computation, and (d) can be applied on any network architecture.
Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning
In this paper we study the problem of learning to learn at both training and test time in the context of visual navigation.
Ranked #2 on
Visual Navigation
on AI2-THOR
From Recognition to Cognition: Visual Commonsense Reasoning
While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world.
Multiple-choice
Multiple Choice Question Answering (MCQA)
+1
Visual Semantic Navigation using Scene Priors
Do we use the semantic/functional priors we have built over years to efficiently search and navigate?
PhotoShape: Photorealistic Materials for Large-Scale Shape Collections
Existing online 3D shape repositories contain thousands of 3D models but lack photorealistic appearance.
Label Refinery: Improving ImageNet Classification through Label Progression
Among the three main components (data, labels, and models) of any supervised learning system, data and models have been the main subjects of active research.
Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
In this paper we describe the egocentric aspect of the dataset and present annotations for Charades-Ego with 68, 536 activity instances in 68. 8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available.
Actor and Observer: Joint Modeling of First and Third-Person Videos
Several theories in cognitive neuroscience suggest that when people interact with the world, or simulate interactions, they do so from a first-person egocentric perspective, and seamlessly transfer knowledge between third-person (observer) and first-person (actor).
Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension
We formalize a new modular variant of current question answering tasks by enforcing complete independence of the document encoder from the question encoder.
Imagine This! Scripts to Compositions to Videos
Imagining a scene described in natural language with realistic layout and appearance of entities is the ultimate test of spatial, visual, and semantic world knowledge.
YOLOv3: An Incremental Improvement
At 320x320 YOLOv3 runs in 22 ms at 28. 2 mAP, as accurate as SSD but three times faster.
One-stage Anchor-free Oriented Object Detection
Real-Time Object Detection
DOCK: Detecting Objects by transferring Common-sense Knowledge
We present a scalable approach for Detecting Objects by transferring Common-sense Knowledge (DOCK) from source to target categories.
Who Let The Dogs Out? Modeling Dog Behavior From Visual Data
We introduce the task of directly modeling a visually intelligent agent.
AI2-THOR: An Interactive 3D Environment for Visual AI
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor. allenai. org.
IQA: Visual Question Answering in Interactive Environments
Our experiments show that our proposed model outperforms popular single controller based methods on IQUAD V1.
Structured Set Matching Networks for One-Shot Part Labeling
Diagrams often depict complex phenomena and serve as a good test bed for visual and textual reasoning.
Neural Speed Reading via Skim-RNN
Inspired by the principles of speed reading, we introduce Skim-RNN, a recurrent neural network (RNN) that dynamically decides to update only a small fraction of the hidden state for relatively unimportant input tokens.
AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Video
This paper describes our approach to detect and to predict natural human arm movements in the future, a key challenge in brain computer interfacing that has never before been attempted.
Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension
Our analysis shows that a significant portion of questions require complex parsing of the text and the diagrams and reasoning, indicating that our dataset is more complex compared to previous machine comprehension and visual question answering datasets.
Visual Semantic Planning using Deep Successor Representations
A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world.
Re3 : Real-Time Recurrent Regression Networks for Visual Tracking of Generic Objects
Robust object tracking requires knowledge and understanding of the object being tracked: its appearance, its motion, and how it changes over time.
SeGAN: Segmenting and Generating the Invisible
Objects often occlude each other in scenes; Inferring their appearance beyond their visible parts plays an important role in scene understanding, depth estimation, object interaction and manipulation.
See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content
Doing so requires estimating the volume of the cup, approximating the amount of water in the pitcher, and predicting the behavior of water when we tilt the pitcher.
YOLO9000: Better, Faster, Stronger
On the 156 classes not in COCO, YOLO9000 gets 16. 0 mAP.
Ranked #4 on
Dense Object Detection
on SKU-110K
Asynchronous Temporal Fields for Action Recognition
Actions are more than just movements and trajectories: we cook to eat and we hold a cup to drink from it.
Ranked #11 on
Action Detection
on Charades
Commonly Uncommon: Semantic Sparsity in Situation Recognition
Semantic sparsity is a common challenge in structured visual classification problems; when the output space is complex, the vast majority of the possible predictions are rarely, if ever, seen in the training set.
LCNN: Lookup-based Convolutional Neural Network
We introduce LCNN, a lookup-based convolutional neural network that encodes convolutions by few lookups to a dictionary that is trained to cover the space of weights in CNNs.
Bidirectional Attention Flow for Machine Comprehension
Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query.
Ranked #4 on
Question Answering
on CNN / Daily Mail
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine.
Much Ado About Time: Exhaustive Annotation of Temporal Data
We conclude that the optimal strategy is to ask as many questions as possible in a HIT (up to 52 binary questions after watching a 30-second video clip in our experiments).
Query-Reduction Networks for Question Answering
In this paper, we study the problem of question answering when reasoning over multiple facts is required.
Ranked #2 on
Question Answering
on bAbi
Situation Recognition: Visual Semantic Role Labeling for Image Understanding
This paper introduces situation recognition, the problem of producing a concise summary of the situation an image depicts including: (1) the main activity (e. g., clipping), (2) the participating actors, objects, substances, and locations (e. g., man, shears, sheep, wool, and field) and most importantly (3) the roles these participants play in the activity (e. g., the man is clipping, the shears are his tool, the wool is being clipped from the sheep, and the clipping is in a field).
Newtonian Scene Understanding: Unfolding the Dynamics of Objects in Static Images
Direct and explicit estimation of the forces and the motion of objects from a single image is extremely challenging.
A Task-Oriented Approach for Cost-Sensitive Recognition
With the recent progress in visual recognition, we have already started to see a surge of vision related real-world applications.
Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks
As 3D movie viewing becomes mainstream and Virtual Reality (VR) market emerges, the demand for 3D contents is growing rapidly.
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
Each video is annotated by multiple free-text descriptions, action labels, action intervals and classes of interacted objects.
A Diagram Is Worth A Dozen Images
We define syntactic parsing of diagrams as learning to infer DPGs for diagrams and study semantic interpretation and reasoning of diagrams in the context of diagram question answering.
"What happens if..." Learning to Predict the Effect of Forces in Images
To build a dataset of forces in scenes, we reconstructed all images in SUN RGB-D dataset in a physics simulator to estimate the physical movements of objects caused by external forces applied to them.
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
We propose two efficient approximations to standard convolutional neural networks: Binary-Weight-Networks and XNOR-Networks.
Are Elephants Bigger than Butterflies? Reasoning about Sizes of Objects
In this paper, we introduce a method to automatically infer object sizes, leveraging visual and textual information from web.
Toward a Taxonomy and Computational Models of Abnormalities in Images
In this paper we study various types of atypicalities in images in a more comprehensive way than has been done before.
Actions ~ Transformations
In this paper, we propose a novel representation for actions by modeling an action as a transformation which changes the state of the environment before the action happens (precondition) to the state after the action (effect).
Generating Notifications for Missing Actions: Don't Forget to Turn the Lights Off!
The overall prediction accuracy is 46. 2% when only 10 frames of an action are seen (2/3 of a sec).
Unsupervised Deep Embedding for Clustering Analysis
Clustering is central to many data-driven application domains and has been studied extensively in terms of distance functions and grouping algorithms.
Ranked #4 on
Image Clustering
on CMU-PIE
Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images
Direct and explicit estimation of the forces and the motion of objects from a single image is extremely challenging.
VISALOGY: Answering Visual Analogy Questions
In this paper, we study the problem of answering visual analogy questions.
Segment-Phrase Table for Semantic Segmentation, Visual Entailment and Paraphrasing
Next, we show that the association of high-quality segmentations to textual phrases aids in richer semantic understanding and reasoning of these textual phrases.
You Only Look Once: Unified, Real-Time Object Detection
A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation.
Ranked #1 on
Real-Time Object Detection
on PASCAL VOC 2007
VisKE: Visual Knowledge Extraction and Question Answering by Visual Verification of Relation Phrases
How can we know whether a statement about our world is valid.
Discriminative and Consistent Similarities in Instance-Level Multiple Instance Learning
In this paper we present a bottom-up method to instance-level Multiple Instance Learning (MIL) that learns to discover positive instances with globally constrained reasoning about local pairwise similarities.
Image Classification and Retrieval from User-Supplied Tags
This paper proposes direct learning of image classification from user-supplied tags, without filtering.
Abnormal Object Recognition: A Comprehensive Study
When describing images, humans tend not to talk about the obvious, but rather mention what they find interesting.
Predicting Failures of Vision Systems
We show that a surprisingly straightforward and general approach, that we call ALERT, can predict the likely accuracy (or failure) of a variety of computer vision systems semantic segmentation, vanishing point and camera parameter estimation, and image memorability prediction on individual input images.
Incorporating Scene Context and Object Layout into Appearance Modeling
In this paper, we propose a method to learn scene structures that can encode three main interlacing components of a scene: the scene category, the context-specific appearance of objects, and their layout.
Learning Everything about Anything: Webly-Supervised Visual Concept Learning
How can we learn a model for any concept that exhaustively covers all its appearance variations, while requiring minimal or no human supervision for compiling the vocabulary of visual variance, gathering the training images and annotations, and learning the models?
Adding Unlabeled Samples to Categories by Learned Attributes
We propose a method to expand the visual coverage of training sets that consist of a small number of labeled examples using learned attributes.
Multi-attribute Queries: To Merge or Not to Merge?
We exploit a discriminative binary space to compute these geometric quantities efficiently.
Object-Centric Anomaly Detection by Attribute-Based Reasoning
When describing images, humans tend not to talk about the obvious, but rather mention what they find interesting.
