Search Results for author:
Found 67 papers, 33 papers with code
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
We present Habitat 3. 0: a simulation platform for studying collaborative human-robot tasks in home environments.
On Evaluation of Embodied Navigation Agents
Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence.
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e. g., find a chair, by navigating to it.
AI2-THOR: An Interactive 3D Environment for Visual AI
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor. allenai. org.
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
We argue that interactive and embodied visual AI has reached a stage of development similar to visual recognition prior to the advent of these ecosystems.
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
In this work, we present CORA, a platform for Continual Reinforcement Learning Agents that provides benchmarks, baselines, and metrics in a single code package.
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks.
AllenAct: A Framework for Embodied AI Research
The domain of Embodied AI, in which agents learn to complete tasks through interaction with their environment from egocentric observations, has experienced substantial growth with the advent of deep reinforcement learning and increased interest from the computer vision, NLP, and robotics communities.
Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning
In this paper we study the problem of learning to learn at both training and test time in the context of visual navigation.
Ranked #2 on
Visual Navigation
on AI2-THOR
Visual Room Rearrangement
We particularly focus on the task of Room Rearrangement: an agent begins by exploring a room and recording objects' initial configurations.
Simple but Effective: CLIP Embeddings for Embodied AI
Contrastive language image pretraining (CLIP) encoders have been shown to be beneficial for a range of visual tasks from classification and detection to captioning and image manipulation.
ManipulaTHOR: A Framework for Visual Object Manipulation
Object manipulation is an established research domain within the robotics community and poses several challenges including manipulator motion, grasping and long-horizon planning, particularly when dealing with oft-overlooked practical setups involving visually rich and complex scenes, manipulation using mobile agents (as opposed to tabletop manipulation), and generalization to unseen environments and objects.
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources.
Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Second
Specifically, a Fetch robot (equipped with a mobile base, 7DoF arm, RGBD camera, egomotion, and onboard sensing) is spawned in a home environment and asked to rearrange objects - by navigating to an object, picking it up, navigating to a target location, and then placing the object at the target location.
Who Let The Dogs Out? Modeling Dog Behavior From Visual Data
We introduce the task of directly modeling a visually intelligent agent.
SeGAN: Segmenting and Generating the Invisible
Objects often occlude each other in scenes; Inferring their appearance beyond their visible parts plays an important role in scene understanding, depth estimation, object interaction and manipulation.
Container: Context Aggregation Network
Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations.
Ranked #462 on
Image Classification
on ImageNet
Container: Context Aggregation Networks
Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations.
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
In contrast to the existing knowledge-based VQA datasets, the questions generally cannot be answered by simply querying a knowledge base, and instead require some form of commonsense reasoning about the scene depicted in the image.
Interactron: Embodied Adaptive Object Detection
Our adaptive object detection model provides a 7. 2 point improvement in AP (and 12. 7 points in AP50) over DETR, a recent, high-performance object detector.
What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision.
Factorizing Perception and Policy for Interactive Instruction Following
Performing simple household tasks based on language directives is very natural to humans, yet it remains an open challenge for AI agents.
Contrasting Contrastive Self-Supervised Representation Learning Pipelines
In the past few years, we have witnessed remarkable breakthroughs in self-supervised representation learning.
Visual Semantic Navigation using Scene Priors
Do we use the semantic/functional priors we have built over years to efficiently search and navigate?
Pushing it out of the Way: Interactive Visual Navigation
In this paper, we study the problem of interactive navigation where agents learn to change the environment to navigate more efficiently to their goals.
Multi-Modal Answer Validation for Knowledge-Based VQA
Instead of searching for the answer in a vast collection of often irrelevant facts as most existing approaches do, MAVEx aims to learn how to extract relevant knowledge from noisy sources, which knowledge source to trust for each answer candidate, and how to validate the candidate using that source.
RobustNav: Towards Benchmarking Robustness in Embodied Navigation
As an attempt towards assessing the robustness of embodied navigation agents, we propose RobustNav, a framework to quantify the performance of embodied navigation agents when exposed to a wide variety of visual - affecting RGB inputs - and dynamics - affecting transition dynamics - corruptions.
Ask4Help: Learning to Leverage an Expert for Embodied Tasks
Embodied AI agents continue to become more capable every year with the advent of new models, environments, and benchmarks, but are still far away from being performant and reliable enough to be deployed in real, user-facing, applications.
What do navigation agents learn about their environment?
We use iSEE to probe the dynamic representations produced by these agents for the presence of information about the agent as well as the environment.
Visual Reaction: Learning to Play Catch with Your Drone
In this paper we address the problem of visual reaction: the task of interacting with dynamic environments where the changes in the environment are not necessarily caused by the agent itself.
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine.
Neural Priming for Sample-Efficient Adaptation
Performing lightweight updates on the recalled data significantly improves accuracy across a variety of distribution shift and transfer learning benchmarks.
Neural Radiance Field Codebooks
Compositional representations of the world are a promising step towards enabling high-level scene understanding and efficient transfer to downstream tasks.
See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content
Doing so requires estimating the volume of the cup, approximating the amount of water in the pitcher, and predicting the behavior of water when we tilt the pitcher.
Visual Semantic Planning using Deep Successor Representations
A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world.
"What happens if..." Learning to Predict the Effect of Forces in Images
To build a dataset of forces in scenes, we reconstructed all images in SUN RGB-D dataset in a physics simulator to estimate the physical movements of objects caused by external forces applied to them.
Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images
Direct and explicit estimation of the forces and the motion of objects from a single image is extremely challenging.
A Coarse-to-Fine Model for 3D Pose Estimation and Sub-category Recognition
Despite the fact that object detection, 3D pose estimation, and sub-category recognition are highly correlated tasks, they are usually addressed independently from each other because of the huge space of parameters.
Human-Machine CRFs for Identifying Bottlenecks in Holistic Scene Understanding
Recent trends in image understanding have pushed for holistic scene understanding models that jointly reason about various tasks such as object detection, scene recognition, shape analysis, contextual reasoning, and local appearance based classifiers.
Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts
Our model automatically decouples the holistic object or body parts from the model when they are hard to detect.
Bottom-Up Segmentation for Top-Down Detection
When employing the parts, we outperform the original DPM [14] in 19 out of 20 classes, achieving an improvement of 8% AP.
Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs
Recent trends in semantic image segmentation have pushed for holistic scene understanding models that jointly reason about various tasks such as object detection, scene recognition, shape analysis, contextual reasoning.
The Role of Context for Object Detection and Semantic Segmentation in the Wild
In this paper we study the role of context in existing state-of-the-art detection and segmentation approaches.
A Task-Oriented Approach for Cost-Sensitive Recognition
With the recent progress in visual recognition, we have already started to see a surge of vision related real-world applications.
Newtonian Scene Understanding: Unfolding the Dynamics of Objects in Static Images
Direct and explicit estimation of the forces and the motion of objects from a single image is extremely challenging.
AAAI-2019 Workshop on Games and Simulations for Artificial Intelligence
This volume represents the accepted submissions from the AAAI-2019 Workshop on Games and Simulations for Artificial Intelligence held on January 29, 2019 in Honolulu, Hawaii, USA.
Complexity of Representation and Inference in Compositional Models with Part Sharing
This paper describes serial and parallel compositional models of multiple objects with part sharing.
Learning Generalizable Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solving, decision making, and socialization.
VisualCOMET: Reasoning about the Dynamic Context of a Still Image
In addition, we provide person-grounding (i. e., co-reference links) between people appearing in the image and people mentioned in the textual commonsense descriptions, allowing for tighter integration between images and text.
Learning About Objects by Learning to Interact with Them
Much of the remarkable progress in computer vision has been focused around fully supervised learning mechanisms relying on highly curated datasets for a variety of tasks.
Learning Flexible Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solving, decision making and socialization.
Learning Visual Representation from Human Interactions
Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision.
Rearrangement: A Challenge for Embodied AI
In the rearrangement task, the goal is to bring a given physical environment into a specified state.
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language.
Hierarchical Modular Framework for Long Horizon Instruction Following
To address such composite tasks, we propose a hierarchical modular approach to learn agents that navigate and manipulate objects in a divide-and-conquer manner for the diverse nature of the entailing tasks.
ASC me to Do Anything: Multi-task Training for Embodied AI
We propose Atomic Skill Completion (ASC), an approach for multi-task training for Embodied AI, where a set of atomic skills shared across multiple tasks are composed together to perform the tasks.
Object Manipulation via Visual Target Localization
Object manipulation is a critical skill required for Embodied AI agents interacting with the world around them.
Continuous Scene Representations for Embodied AI
Our method captures feature relationships between objects, composes them into a graph structure on-the-fly, and situates an embodied agent within the representation.
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding.
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing.
Ranked #1 on
Object Segmentation
on GRIT
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
Navigating to Objects Specified by Images
Our modular method solves sub-tasks of exploration, goal instance re-identification, goal localization, and local navigation.
ENTL: Embodied Navigation Trajectory Learner
ENTL's generic architecture enables sharing of the spatio-temporal sequence encoder for multiple challenging embodied tasks.
Moving Forward by Moving Backward: Embedding Action Impact over Action Semantics
A common assumption when training embodied agents is that the impact of taking an action is stable; for instance, executing the "move ahead" action will always move the agent forward by a fixed distance, perhaps with some small amount of actuator-induced noise.
HomeRobot: Open-Vocabulary Mobile Manipulation
HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks.
Controllable Human-Object Interaction Synthesis
Naively applying a diffusion model fails to predict object motion aligned with the input waypoints and cannot ensure the realism of interactions that require precise hand-object contact and appropriate contact grounded by the floor.
GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation
The Embodied AI community has made significant strides in visual navigation tasks, exploring targets from 3D coordinates, objects, language descriptions, and images.
