Search Results for author:
Found 84 papers, 37 papers with code
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition
We analyze the suitability of our new primitive for video action recognition and explore several novel variations of our approach to enable stronger representational flexibility while maintaining an efficient design.
PRIME: Scaffolding Manipulation Tasks with Behavior Primitives for Data-Efficient Imitation Learning
Imitation learning has shown great potential for enabling robots to acquire complex manipulation behaviors.
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
In each iteration, the image is annotated with a visual representation of proposals that the VLM can refer to (e. g., candidate robot actions, localizations, or trajectories).
Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
Two desiderata of reinforcement learning (RL) algorithms are the ability to learn from relatively little experience and the ability to learn policies that generalize to a range of problem specifications.
Edge Wasserstein Distance Loss for Oriented Object Detection
Differing from exploited the Gaussian distribution to get analytical form of distance measure, we propose a novel oriented regression loss, Wasserstein Distance(EWD) loss, to alleviate the square-like problem.
LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discovery
We introduce LOTUS, a continual imitation learning algorithm that empowers a physical robot to continuously and efficiently learn to solve new manipulation tasks throughout its lifespan.
Model-Based Runtime Monitoring with Interactive Imitation Learning
Unlike prior work that cannot foresee future failures or requires failure experiences for training, our method learns a latent-space dynamics model and a failure classifier, enabling our method to simulate future action outcomes and detect out-of-distribution and high-risk states preemptively.
Interactive Robot Learning from Verbal Correction
A key feature of OLAF is its ability to update the robot's visuomotor neural policy based on the verbal feedback to avoid repeating mistakes in the future.
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Imitation learning from a large set of human demonstrations has proved to be an effective paradigm for building capable robot agents.
Learning Generalizable Manipulation Policies with Object-Centric 3D Representations
We introduce GROOT, an imitation learning method for learning robust policies with object-centric and 3D priors.
Eureka: Human-Level Reward Design via Coding Large Language Models
The generality of Eureka also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating.
AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents
We introduce AMAGO, an in-context Reinforcement Learning (RL) agent that uses sequence models to tackle the challenges of generalization, long-term memory, and meta-learning.
Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flow
Incorporating semantic priors with self-supervised flow training, Doduo produces accurate dense correspondence robust to the dynamic changes of the scenes.
Granger-Causal Hierarchical Skill Discovery
Reinforcement Learning (RL) has demonstrated promising results in learning policies for complex tasks, but it often suffers from low sample efficiency and limited transferability.
Voyager: An Open-Ended Embodied Agent with Large Language Models
We introduce Voyager, the first LLM-powered embodied lifelong learning agent in Minecraft that continuously explores the world, acquires diverse skills, and makes novel discoveries without human intervention.
Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids
To apply this representation to monocular scene reconstruction, we develop a scale calibration algorithm for fast geometric initialization from monocular depth priors.
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Augmenting pretrained language models (LMs) with a vision encoder (e. g., Flamingo) has obtained the state-of-the-art results in image-to-text generation.
Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception
We demonstrate the effectiveness of our approach in both simulation and real-world scenes.
Few-View Object Reconstruction with Unknown Categories and Camera Poses
The reconstruction results under predicted poses are comparable to the ones using ground-truth poses.
Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment
To harness the capabilities of state-of-the-art robot learning models while embracing their imperfections, we present Sirius, a principled framework for humans and robots to collaborate through a division of work.
Learning and Retrieval from Prior Data for Skill-based Imitation Learning
Imitation learning offers a promising path for robots to learn general-purpose behaviors, but traditionally has exhibited limited scalability due to high data supervision requirements and brittle generalization.
VIMA: General Robot Manipulation with Multimodal Prompts
We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens.
Learning to Walk by Steering: Perceptive Quadrupedal Locomotion in Dynamic Environments
We present a hierarchical learning framework, named PRELUDE, which decomposes the problem of perceptive locomotion into high-level decision-making to predict navigation commands and low-level gait generation to realize the target commands.
Causal Dynamics Learning for Task-Independent State Abstraction
Learning dynamics models accurately is an important goal for Model-Based Reinforcement Learning (MBRL), but most MBRL methods learn a dense dynamics model which is vulnerable to spurious correlations and therefore generalizes poorly to unseen states.
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Autonomous agents have made great strides in specialist domains like Atari games and Go.
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
A significant gap remains between today's visual pattern recognition models and human-level visual cognition especially when it comes to few-shot learning and compositional reasoning of novel concepts.
Ranked #1 on
Few-Shot Image Classification
on Bongard-HOI
COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehicles
To evaluate our model, we develop AutoCastSim, a network-augmented driving simulation framework with example accident-prone scenarios.
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i. e., systematic generalization.
Ranked #1 on
Zero-Shot Human-Object Interaction Detection
on HICO
ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation
Manipulating volumetric deformable objects in the real world, like plush toys and pizza dough, bring substantial challenges due to infinite shape variations, non-rigid motions, and partial observability.
Ditto: Building Digital Twins of Articulated Objects from Interaction
We also apply Ditto to real-world objects and deploy the recreated digital twins in physical simulation.
Pre-Trained Language Models for Interactive Decision-Making
Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.
Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
However, these approaches require larger state distributions to be covered as more policies are sequenced, and thus are limited to short skill sequences.
Reinforcement Learning in Factored Action Spaces using Tensor Decompositions
We present an extended abstract for the previously published work TESSERACT [Mahajan et al., 2021], which proposes a novel solution for Reinforcement Learning (RL) in large, factored action spaces using tensor decompositions.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
Realistic manipulation tasks require a robot to interact with an environment with a prolonged sequence of motor actions.
OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation
Operational Space Control (OSC) has been used as an effective task-space controller for manipulation.
Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation
From the task structures of multi-task demonstrations, we identify skills based on the recurring patterns and train goal-conditioned sensorimotor policies with hierarchical imitation learning.
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation.
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas.
Discovering Generalizable Skills via Automated Generation of Diverse Tasks
To encourage generalizable skills to emerge, our method trains each skill to specialize in the paired task and maximizes the diversity of the generated tasks.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
A student network then learns to mimic the expert policy by supervised learning with strong augmentations, making its representation more robust against visual variations compared to the expert.
Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning
Algorithms derived from Tesseract decompose the Q-tensor across agents and utilise low-rank tensor approximations to model agent interactions relevant to the task.
Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition
Specifically, we 1) adopt the attention mechanism for both the coach and the players; 2) propose a variational objective to regularize learning; and 3) design an adaptive communication method to let the coach decide when to communicate with the players.
Multi-agent Reinforcement Learning
reinforcement-learning
+3
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision
We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision.
Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations
The experimental results in simulation and on the real robot have demonstrated that the use of implicit neural representations and joint learning of grasp affordance and 3D reconstruction have led to state-of-the-art grasping results.
Dynamic Metric Learning: Towards a Scalable Metric Space to Accommodate Multiple Semantic Scales
%We argue that such flexibility is also important for deep metric learning, because different visual concepts indeed correspond to different semantic scales.
Ranked #2 on
Metric Learning
on DyML-Animal
A Coach-Player Framework for Dynamic Team Composition
The performance of our method is comparable or even better than the setting where all players have a full view of the environment, but no coach.
Emergent Hand Morphology and Control from Optimizing Robust Grasps of Diverse Objects
Through experimentation and comparative study, we demonstrate the effectiveness of our approach in discovering robust and cost-efficient hand morphologies for grasping novel objects.
Human-in-the-Loop Imitation Learning using Remote Teleoperation
We develop a simple and effective algorithm to train the policy iteratively on new data collected by the system that encourages the policy to learn how to traverse bottlenecks through the interventions.
Learning Multi-Arm Manipulation Through Collaborative Teleoperation
To address these challenges, we present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks.
Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
Using sensor data from multiple modalities presents an opportunity to encode redundant and complementary features that can be useful when one modality is corrupted or noisy.
Fast Uncertainty Quantification for Deep Object Pose Estimation
Deep learning-based object pose estimators are often unreliable and overconfident especially when the input image is outside the training domain, for instance, with sim2real transfer.
Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning
Inspired by the original one hundred BPs, we propose a new benchmark Bongard-LOGO for human-level concept learning and reasoning.
robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
robosuite is a simulation framework for robot learning powered by the MuJoCo physics engine.
Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion
We present a hierarchical framework that combines model-based control and reinforcement learning (RL) to synthesize robust controllers for a quadruped (the Unitree Laikago).
OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
We propose a variational inference framework OCEAN to perform online task inference for compositional tasks.
Spherical Feature Transform for Deep Metric Learning
Consequently, the feature transform is performed by a rotation that respects the spherical data distributions.
Adaptive Procedural Task Generation for Hard-Exploration Problems
To enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balancing the agent's performance in the generated tasks and the similarity to the target tasks.
Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity
We evaluate the quality of our platform, the diversity of demonstrations in our dataset, and the utility of our dataset via quantitative and qualitative analysis.
Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
The fundamental challenge of planning for multi-step manipulation is to find effective and plausible action sequences that lead to the task goal.
KETO: Learning Keypoint Representations for Tool Manipulation
For this purpose, we present KETO, a framework of learning keypoint representations of tool-based manipulation.
Robotics
6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints
We present 6-PACK, a deep learning approach to category-level 6D object pose tracking on RGB-D data.
Ranked #1 on
6D Pose Estimation using RGBD
on REAL275
(Rerr metric)
Causal Induction from Visual Observations for Goal Directed Tasks
Causal reasoning has been an indispensable capability for humans and other intelligent animals to interact with the physical world.
DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs
A major difficulty of solving continuous POMDPs is to infer the multi-modal distribution of the unobserved true states and to make the planning algorithm dependent on the perceived uncertainty.
Regression Planning Networks
Recent learning-to-plan methods have shown promising results on planning directly from observation space.
SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
We present an overview of SURREAL-System, a reproducible, flexible, and scalable framework for distributed reinforcement learning (RL).
Situational Fusion of Visual Representation for Visual Navigation
A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities.
Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning
The key technical challenge is that the symbol grounding is prone to error with limited training data and leads to subsequent symbolic planning failures.
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback.
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources.
Ranked #4 on
6D Pose Estimation
on LineMOD
RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
Imitation Learning has empowered recent advances in learning robotic manipulation tasks by addressing shortcomings of Reinforcement Learning such as exploration and reward specification.
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback.
Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model.
Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision
We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering.
Reinforcement and Imitation Learning for Diverse Visuomotor Skills
We propose a model-free deep reinforcement learning method that leverages a small amount of demonstration data to assist a reinforcement learning agent.
AI2-THOR: An Interactive 3D Environment for Visual AI
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor. allenai. org.
Neural Task Programming: Learning to Generalize Across Hierarchical Tasks
In this work, we propose a novel robot learning framework called Neural Task Programming (NTP), which bridges the idea of few-shot learning from demonstration and neural program induction.
Knowledge Acquisition for Visual Question Answering via Iterative Querying
Humans possess an extraordinary ability to learn new skills and new knowledge for problem solving.
Visual Semantic Planning using Deep Successor Representations
A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world.
Scene Graph Generation by Iterative Message Passing
In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image.
Ranked #9 on
Panoptic Scene Graph Generation
on PSG Dataset
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine.
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering.
Visual7W: Grounded Question Answering in Images
It enables a new type of QA with visual answers, in addition to textual answers used in previous work.
Multiple-choice
Multiple Choice Question Answering (MCQA)
+2
Action Recognition by Hierarchical Mid-level Action Elements
Realistic videos of human actions exhibit rich spatiotemporal structures at multiple levels of granularity: an action can always be decomposed into multiple finer-grained elements in both space and time.
Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
The complexity of the visual world creates significant challenges for comprehensive visual understanding.
