Search Results for author:
Found 60 papers, 33 papers with code
OBJECT 3DIT: Language-guided 3D-aware Image Editing
Existing image editing tools, while powerful, typically disregard the underlying 3D geometry from which the image is projected.
Objaverse-XL: A Universe of 10M+ 3D Objects
In this work, we present Objaverse-XL, a dataset of over 10 million 3D objects.
MIMIC: Masked Image Modeling with Image Correspondences
Unfortunately, the effective pretraining datasets are those with multi-view scenes and have only been curated using annotated 3D meshes, point clouds, and camera parameters from simulated environments.
SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
In the last year alone, a surge of new benchmarks to measure compositional understanding of vision-language models have permeated the machine learning ecosystem.
Neural Priming for Sample-Efficient Adaptation
Performing lightweight updates on the recalled data significantly improves accuracy across a variety of distribution shift and transfer learning benchmarks.
Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models
Inconsistent AI models are considered brittle and untrustworthy by human users and are more challenging to incorporate into larger systems that take dependencies on their outputs.
Neural Radiance Field Codebooks
Compositional representations of the world are a promising step towards enabling high-level scene understanding and efficient transfer to downstream tasks.
EXCALIBUR: Encouraging and Evaluating Embodied Exploration
Humans constantly explore and learn about their environment out of curiosity, gather information, and update their models of the world.
Objaverse: A Universe of Annotated 3D Objects
Massive data corpora like WebText, Wikipedia, Conceptual Captions, WebImageText, and LAION have propelled recent dramatic progress in AI.
Phone2Proc: Bringing Robust Robots Into Our Chaotic World
Training embodied agents in simulation has become mainstream for the embodied AI community.
A General Purpose Supervisory Signal for Embodied Agents
Training effective embodied AI agents often involves manual reward engineering, expert imitation, specialized components such as maps, or leveraging additional sensors for depth and localization.
SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding
Remote sensing images are useful for a wide variety of planet monitoring applications, from tracking deforestation to tackling illegal fishing.
Visual Programming: Compositional visual reasoning without training
We present VISPROG, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions.
I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision
We produce models using only text training data on four representative tasks: image captioning, visual entailment, visual question answering and visual news captioning, and evaluate them on standard benchmarks using images.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
What do navigation agents learn about their environment?
We use iSEE to probe the dynamic representations produced by these agents for the presence of information about the agent as well as the environment.
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing.
Ranked #1 on
Object Segmentation
on GRIT
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding.
GRIT: General Robust Image Task Benchmark
Computer vision models excel at making predictions when the test distribution closely resembles the training distribution.
Object Manipulation via Visual Target Localization
Object manipulation is a critical skill required for Embodied AI agents interacting with the world around them.
ASC me to Do Anything: Multi-task Training for Embodied AI
We propose Atomic Skill Completion (ASC), an approach for multi-task training for Embodied AI, where a set of atomic skills shared across multiple tasks are composed together to perform the tasks.
Webly Supervised Concept Expansion for General Purpose Vision Models
This work presents an effective and inexpensive alternative: learn skills from supervised datasets, learn concepts from web image search, and leverage a key characteristic of GPVs: the ability to transfer visual knowledge across skills.
Ranked #2 on
Visual Question Answering (VQA)
on GRIT
Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture
To reduce the time and expertise required to develop new applications, we would like to create general purpose vision systems that can learn and perform a range of tasks without any modification to the architecture or learning process.
Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text
We investigate these challenges in the context of Iconary, a collaborative game of drawing and guessing based on Pictionary, that poses a novel challenge for the research community.
Container: Context Aggregation Networks
Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations.
Simple but Effective: CLIP Embeddings for Embodied AI
Contrastive language image pretraining (CLIP) encoders have been shown to be beneficial for a range of visual tasks from classification and detection to captioning and image manipulation.
RobustNav: Towards Benchmarking Robustness in Embodied Navigation
As an attempt towards assessing the robustness of embodied navigation agents, we propose RobustNav, a framework to quantify the performance of embodied navigation agents when exposed to a wide variety of visual - affecting RGB inputs - and dynamics - affecting transition dynamics - corruptions.
Container: Context Aggregation Network
Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations.
Ranked #430 on
Image Classification
on ImageNet
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language.
ManipulaTHOR: A Framework for Visual Object Manipulation
Object manipulation is an established research domain within the robotics community and poses several challenges including manipulator motion, grasping and long-horizon planning, particularly when dealing with oft-overlooked practical setups involving visually rich and complex scenes, manipulation using mobile agents (as opposed to tabletop manipulation), and generalization to unseen environments and objects.
GridToPix: Training Embodied Agents with Minimal Supervision
While deep reinforcement learning (RL) promises freedom from hand-labeled data, great successes, especially for Embodied AI, require significant work to create supervision via carefully shaped rewards.
Visual Semantic Role Labeling for Video Understanding
We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling.
Towards General Purpose Vision Systems
To reduce the time and expertise required to develop new applications, we would like to create general purpose vision systems that can learn and perform a range of tasks without any modification to the architecture or learning process.
Visual Room Rearrangement
We particularly focus on the task of Room Rearrangement: an agent begins by exploring a room and recording objects' initial configurations.
Learning Flexible Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solving, decision making and socialization.
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
X-LXMERT's image generation capabilities rival state of the art generative models while its question answering and captioning abilities remains comparable to LXMERT.
AllenAct: A Framework for Embodied AI Research
The domain of Embodied AI, in which agents learn to complete tasks through interaction with their environment from egocentric observations, has experienced substantial growth with the advent of deep reinforcement learning and increased interest from the computer vision, NLP, and robotics communities.
Bridging the Imitation Gap by Adaptive Insubordination
However, we show that when the teaching agent makes decisions with access to privileged information that is unavailable to the student, this information is marginalized during imitation learning, resulting in an "imitation gap" and, potentially, poor results.
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks
Autonomous agents must learn to collaborate.
FLUID: A Unified Evaluation Framework for Flexible Sequential Data
To foster research towards the goal of general ML methods, we introduce a new unified evaluation framework - FLUID (Flexible Sequential Data).
Supermasks in Superposition
We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting.
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e. g., find a chair, by navigating to it.
Feel The Music: Automatically Generating A Dance For An Input Song
We encode intuitive, flexible heuristics for what a 'good' dance is: the structure of the dance should align with the structure of the music.
Learning About Objects by Learning to Interact with Them
Much of the remarkable progress in computer vision has been focused around fully supervised learning mechanisms relying on highly curated datasets for a variety of tasks.
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
We argue that interactive and embodied visual AI has reached a stage of development similar to visual recognition prior to the advent of these ecosystems.
Grounded Situation Recognition
We introduce Grounded Situation Recognition (GSR), a task that requires producing structured semantic summaries of images describing: the primary activity, entities engaged in the activity with their roles (e. g. agent, tool), and bounding-box groundings of entities.
Ranked #5 on
Situation Recognition
on imSitu
Learning Generalizable Visual Representations via Interactive Gameplay
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solving, decision making, and socialization.
What's Hidden in a Randomly Weighted Neural Network?
Training a neural network is synonymous with learning the values of the weights.
Ranked #868 on
Image Classification
on ImageNet
Two Body Problem: Collaborative Visual Task Completion
Collaboration is a necessary skill to perform tasks that are beyond one agent's capabilities.
ELASTIC: Improving CNNs with Dynamic Scaling Policies
We formulate the scaling policy as a non-linear function inside the network's structure that (a) is learned from data, (b) is instance specific, (c) does not add extra computation, and (d) can be applied on any network architecture.
Imagine This! Scripts to Compositions to Videos
Imagining a scene described in natural language with realistic layout and appearance of entities is the ultimate test of spatial, visual, and semantic world knowledge.
AI2-THOR: An Interactive 3D Environment for Visual AI
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor. allenai. org.
IQA: Visual Question Answering in Interactive Environments
Our experiments show that our proposed model outperforms popular single controller based methods on IQUAD V1.
Structured Set Matching Networks for One-Shot Part Labeling
Diagrams often depict complex phenomena and serve as a good test bed for visual and textual reasoning.
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively).
Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension
Our analysis shows that a significant portion of questions require complex parsing of the text and the diagrams and reasoning, indicating that our dataset is more complex compared to previous machine comprehension and visual question answering datasets.
C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
Finally, we evaluate several existing VQA models under this new setting and show that the performances of these models degrade by a significant amount compared to the original VQA setting.
Bidirectional Attention Flow for Machine Comprehension
Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query.
Ranked #4 on
Question Answering
on CNN / Daily Mail
Semantic Parsing to Probabilistic Programs for Situated Question Answering
Situated question answering is the problem of answering questions about an environment such as an image or diagram.
A Diagram Is Worth A Dozen Images
We define syntactic parsing of diagrams as learning to infer DPGs for diagrams and study semantic interpretation and reasoning of diagrams in the context of diagram question answering.
