Search Results for author:
Found 51 papers, 26 papers with code
Continuous Scene Representations for Embodied AI
Our method captures feature relationships between objects, composes them into a graph structure on-the-fly, and situates an embodied agent within the representation.
Object Manipulation via Visual Target Localization
Object manipulation is a critical skill required for Embodied AI agents interacting with the world around them.
ASC me to Do Anything: Multi-task Training for Embodied AI
We propose Atomic Skill Completion (ASC), an approach for multi-task training for Embodied AI, where a set of atomic skills shared across multiple tasks are composed together to perform the tasks.
Interactron: Embodied Adaptive Object Detection
Moreover, we show that our object detection model adapts to environments with completely different appearance characteristics, and its performance is on par with a model trained with full supervision for those environments.
Container: Context Aggregation Networks
Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations.
Simple but Effective: CLIP Embeddings for Embodied AI
Contrastive language image pretraining (CLIP) encoders have been shown to be beneficial for a range of visual tasks from classification and detection to captioning and image manipulation.
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
In this work, we present CORA, a platform for Continual Reinforcement Learning Agents that provides benchmarks, baselines, and metrics in a single code package.
Hierarchical Modular Framework for Long Horizon Instruction Following
To address such composite tasks, we propose a hierarchical modular approach to learn agents that navigate and manipulate objects in a divide-and-conquer manner for the diverse nature of the entailing tasks.
RobustNav: Towards Benchmarking Robustness in Embodied Navigation
As an attempt towards assessing the robustness of embodied navigation agents, we propose RobustNav, a framework to quantify the performance of embodied navigation agents when exposed to a wide variety of visual - affecting RGB inputs - and dynamics - affecting transition dynamics - corruptions.
Container: Context Aggregation Network
Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations.
Ranked #252 on
Image Classification
on ImageNet
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language.
Pushing it out of the Way: Interactive Visual Navigation
In this paper, we study the problem of interactive navigation where agents learn to change the environment to navigate more efficiently to their goals.
ManipulaTHOR: A Framework for Visual Object Manipulation
Object manipulation is an established research domain within the robotics community and poses several challenges including manipulator motion, grasping and long-horizon planning, particularly when dealing with oft-overlooked practical setups involving visually rich and complex scenes, manipulation using mobile agents (as opposed to tabletop manipulation), and generalization to unseen environments and objects.
Visual Room Rearrangement
We particularly focus on the task of Room Rearrangement: an agent begins by exploring a room and recording objects' initial configurations.
Contrasting Contrastive Self-Supervised Representation Learning Pipelines
In the past few years, we have witnessed remarkable breakthroughs in self-supervised representation learning.
Multi-Modal Answer Validation for Knowledge-Based VQA
Instead of searching for the answer in a vast collection of often irrelevant facts as most existing approaches do, MAVEx aims to learn how to extract relevant knowledge from noisy sources, which knowledge source to trust for each answer candidate, and how to validate the candidate using that source.
Learning Visual Representation from Human Interactions
Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision.
Learning Flexible Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solving, decision making and socialization.
Factorizing Perception and Policy for Interactive Instruction Following
Performing simple household tasks based on language directives is very natural to humans, yet it remains an open challenge for AI agents.
Rearrangement: A Challenge for Embodied AI
In the rearrangement task, the goal is to bring a given physical environment into a specified state.
What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision.
AllenAct: A Framework for Embodied AI Research
The domain of Embodied AI, in which agents learn to complete tasks through interaction with their environment from egocentric observations, has experienced substantial growth with the advent of deep reinforcement learning and increased interest from the computer vision, NLP, and robotics communities.
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e. g., find a chair, by navigating to it.
Learning About Objects by Learning to Interact with Them
Much of the remarkable progress in computer vision has been focused around fully supervised learning mechanisms relying on highly curated datasets for a variety of tasks.
VisualCOMET: Reasoning about the Dynamic Context of a Still Image
In addition, we provide person-grounding (i. e., co-reference links) between people appearing in the image and people mentioned in the textual commonsense descriptions, allowing for tighter integration between images and text.
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
We argue that interactive and embodied visual AI has reached a stage of development similar to visual recognition prior to the advent of these ecosystems.
Learning Generalizable Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solving, decision making, and socialization.
Visual Reaction: Learning to Play Catch with Your Drone
In this paper we address the problem of visual reaction: the task of interacting with dynamic environments where the changes in the environment are not necessarily caused by the agent itself.
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks.
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources.
AAAI-2019 Workshop on Games and Simulations for Artificial Intelligence
This volume represents the accepted submissions from the AAAI-2019 Workshop on Games and Simulations for Artificial Intelligence held on January 29, 2019 in Honolulu, Hawaii, USA.
Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning
In this paper we study the problem of learning to learn at both training and test time in the context of visual navigation.
Ranked #2 on
Visual Navigation
on AI2-THOR
Visual Semantic Navigation using Scene Priors
Do we use the semantic/functional priors we have built over years to efficiently search and navigate?
On Evaluation of Embodied Navigation Agents
Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence.
Who Let The Dogs Out? Modeling Dog Behavior From Visual Data
We introduce the task of directly modeling a visually intelligent agent.
AI2-THOR: An Interactive 3D Environment for Visual AI
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor. allenai. org.
Visual Semantic Planning using Deep Successor Representations
A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world.
SeGAN: Segmenting and Generating the Invisible
Objects often occlude each other in scenes; Inferring their appearance beyond their visible parts plays an important role in scene understanding, depth estimation, object interaction and manipulation.
See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content
Doing so requires estimating the volume of the cup, approximating the amount of water in the pitcher, and predicting the behavior of water when we tilt the pitcher.
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine.
A Task-Oriented Approach for Cost-Sensitive Recognition
With the recent progress in visual recognition, we have already started to see a surge of vision related real-world applications.
Newtonian Scene Understanding: Unfolding the Dynamics of Objects in Static Images
Direct and explicit estimation of the forces and the motion of objects from a single image is extremely challenging.
"What happens if..." Learning to Predict the Effect of Forces in Images
To build a dataset of forces in scenes, we reconstructed all images in SUN RGB-D dataset in a physics simulator to estimate the physical movements of objects caused by external forces applied to them.
Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images
Direct and explicit estimation of the forces and the motion of objects from a single image is extremely challenging.
A Coarse-to-Fine Model for 3D Pose Estimation and Sub-category Recognition
Despite the fact that object detection, 3D pose estimation, and sub-category recognition are highly correlated tasks, they are usually addressed independently from each other because of the huge space of parameters.
Human-Machine CRFs for Identifying Bottlenecks in Holistic Scene Understanding
Recent trends in image understanding have pushed for holistic scene understanding models that jointly reason about various tasks such as object detection, scene recognition, shape analysis, contextual reasoning, and local appearance based classifiers.
Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts
Our model automatically decouples the holistic object or body parts from the model when they are hard to detect.
The Role of Context for Object Detection and Semantic Segmentation in the Wild
In this paper we study the role of context in existing state-of-the-art detection and segmentation approaches.
Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs
Recent trends in semantic image segmentation have pushed for holistic scene understanding models that jointly reason about various tasks such as object detection, scene recognition, shape analysis, contextual reasoning.
Bottom-Up Segmentation for Top-Down Detection
When employing the parts, we outperform the original DPM [14] in 19 out of 20 classes, achieving an improvement of 8% AP.
Complexity of Representation and Inference in Compositional Models with Part Sharing
This paper describes serial and parallel compositional models of multiple objects with part sharing.
