Search Results for author:
Found 45 papers, 14 papers with code
SPRINT: Scalable Policy Pre-Training via Language Instruction Relabeling
Pre-training robot policies with a rich set of skills can substantially accelerate the learning of downstream tasks.
FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation
We benchmark the performance of offline RL and IL algorithms on our assembly tasks and demonstrate the need to improve such algorithms to be able to solve our tasks in the real world, providing ample opportunities for future research.
Hierarchical Neural Program Synthesis
Then, we train a high-level module to comprehend the task specification (e. g., input/output pairs or demonstrations) from long programs and produce a sequence of task embeddings, which are then decoded by the program decoder and composed to yield the synthesized program.
Efficient Multi-Task Reinforcement Learning via Selective Behavior Sharing
The ability to leverage shared behaviors between tasks is critical for sample-efficient multi-task reinforcement learning (MTRL).
Cross-Domain Transfer via Semantic Skill Imitation
We propose an approach for semantic imitation, which uses demonstrations from a source domain, e. g. human videos, to accelerate reinforcement learning (RL) in a different target domain, e. g. a robotic manipulator in a simulated kitchen.
PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection
Large-scale data is an essential component of machine learning as demonstrated in recent advances in natural language processing and computer vision research.
Skill-based Model-based Reinforcement Learning
From this intuition, we propose a Skill-based Model-based RL framework (SkiMo) that enables planning in the skill space using a skill dynamics model, which directly predicts the skill outcomes, rather than predicting all small details in the intermediate states, step by step.
Model-based Reinforcement Learning
reinforcement-learning
+1
Task-Induced Representation Learning
We investigate the effectiveness of unsupervised and task-induced representation learning approaches on four visually complex environments, from Distracting DMControl to the CARLA driving simulator.
Generalizable Imitation Learning from Observation via Inferring Goal Proximity
Task progress is intuitive and readily available task information that can guide an agent closer to the desired goal.
Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
However, these approaches require larger state distributions to be covered as more policies are sequenced, and thus are limited to short skill sequences.
Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation
Learning complex manipulation tasks in realistic, obstructed environments is a challenging problem due to hard exploration in the presence of obstacles and high-dimensional visual observations.
Learning to Synthesize Programs as Interpretable and Generalizable Policies
To alleviate the difficulty of learning to compose programs to induce the desired agent behavior from scratch, we propose to first learn a program embedding space that continuously parameterizes diverse behaviors in an unsupervised manner and then search over the learned program embedding space to yield a program that maximizes the return for a given task.
Demonstration-Guided Reinforcement Learning with Learned Skills
Prior approaches for demonstration-guided RL treat every new task as an independent learning problem and attempt to follow the provided demonstrations step-by-step, akin to a human trying to imitate a completely unseen behavior by following the demonstrator's exact muscle movements.
Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding
In this paper, we propose a novel policy transfer method with iterative "environment grounding", IDAPT, that alternates between (1) directly minimizing both visual and dynamics domain gaps by grounding the source environment in the target environment domains, and (2) training a policy on the grounded source environment.
Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning
Active learning is widely used to reduce labeling effort and training time by repeatedly querying only the most beneficial samples from unlabeled data.
Generalization to New Actions in Reinforcement Learning
A fundamental trait of intelligence is the ability to achieve goals in the face of novel circumstances, such as making decisions from new action choices.
Accelerating Reinforcement Learning with Learned Skill Priors
We validate our approach, SPiRL (Skill-Prior RL), on complex navigation and robotic manipulation tasks and show that learned skill priors are essential for effective skill transfer from rich datasets.
Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments
In contrast, motion planners use explicit models of the agent and environment to plan collision-free paths to faraway goals, but suffer from inaccurate models in tasks that require contacts with the environment.
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification
When mastering a complex manipulation task, humans often decompose the task into sub-skills of their body parts, practice the sub-skills independently, and then execute the sub-skills together.
Program Guided Agent
Developing agents that can learn to follow natural language instructions has been an emerging research direction.
To Follow or not to Follow: Selective Imitation Learning from Observations
Learning from demonstrations is a useful way to transfer a skill from one agent to another.
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks
The IKEA Furniture Assembly Environment is one of the first benchmarks for testing and accelerating the automation of complex manipulation tasks.
Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
Model-agnostic meta-learners aim to acquire meta-learned parameters from similar tasks to adapt to novel tasks from the same distribution with few gradient updates.
Keyframing the Future: Discovering Temporal Hierarchy with Keyframe-Inpainter Prediction
To flexibly and efficiently reason about temporal sequences, abstract representations that compactly represent the important information in the sequence are needed.
Generalizing Reinforcement Learning to Unseen Actions
Hence, we propose a framework to enable generalization over both these aspects: understanding an action’s functionality, and using actions to solve tasks through reinforcement learning.
Composing Complex Skills by Learning Transition Policies with Proximity Reward Induction
Intelligent creatures acquire complex skills by exploiting previously learned skills and learning to transition between them.
Learning from Noisy Demonstration Sets via Meta-Learned Suitability Assessor
A noisy and diverse demonstration set may hinder the performances of an agent aiming to acquire certain skills via imitation learning.
Toward Multimodal Model-Agnostic Meta-Learning
One important limitation of such frameworks is that they seek a common initialization shared across the entire task distribution, substantially limiting the diversity of the task distributions that they are able to learn from.
Multi-view to Novel view: Synthesizing Novel Views with Self-Learned Confidence
We address the task of multi-view novel view synthesis, where we are interested in synthesizing a target image with an arbitrary camera pose from given source images.
Ranked #1 on
Novel View Synthesis
on KITTI Novel View Synthesis
Simulator Predictive Control: Using Learned Task Representations and MPC for Zero-Shot Generalization and Sequencing
We complete unseen tasks by choosing new sequences of skill latents to control the robot using MPC, where our MPC model is composed of the pre-trained skill policy executed in the simulation environment, run in parallel with the real robot.
Auto-conditioned Recurrent Mixture Density Networks for Learning Generalizable Robot Skills
Personal robots assisting humans must perform complex manipulation tasks that are typically difficult to specify in traditional motion planning pipelines, where multiple objectives must be met and the high-level context be taken into consideration.
Model-Agnostic Meta-Learning for Multimodal Task Distributions
In this paper, we augment MAML with the capability to identify tasks sampled from a multimodal task distribution and adapt quickly through gradient updates.
Scaling simulation-to-real transfer by learning composable robot skills
In particular, we first use simulation to jointly learn a policy for a set of low-level skills, and a "skill embedding" parameterization which can be used to compose them.
Demo2Vec: Reasoning Object Affordances From Online Videos
Watching expert demonstrations is an important way for humans and robots to reason about affordances of unseen objects.
Ranked #2 on
Video-to-image Affordance Grounding
on OPRA (28x28)
Learning and Using the Arrow of Time
We seek to understand the arrow of time in videos -- what makes videos look like they are playing forwards or backwards?
Ranked #49 on
Self-Supervised Action Recognition
on UCF101
Self-Supervised Action Recognition
Temporal Action Localization
+1
3D Interpreter Networks for Viewer-Centered Wireframe Modeling
3D-INN is trained on real images to estimate 2D keypoint heatmaps from an input image; it then predicts 3D object structure from heatmaps using knowledge learned from synthetic 3D shapes.
Universal Agent for Disentangling Environments and Tasks
Recent state-of-the-art reinforcement learning algorithms are trained under the goal of excelling in one specific task.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Knowledge Acquisition for Visual Question Answering via Iterative Querying
Humans possess an extraordinary ability to learn new skills and new knowledge for problem solving.
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
We propose an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity (e. g., "dressing") to the action (e. g., "mix yogurt") that produced it.
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine.
Single Image 3D Interpreter Network
In this work, we propose 3D INterpreter Network (3D-INN), an end-to-end framework which sequentially estimates 2D keypoint heatmaps and 3D object structure, trained on both real 2D-annotated images and synthetic 3D data.
Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning
Humans demonstrate remarkable abilities to predict physical events in dynamic scenes, and to infer the physical properties of objects from static images.
Discovering States and Transformations in Image Collections
Our system works by generalizing across object classes: states and transformations learned on one set of objects are used to interpret the image collection for an entirely new object class.
Looking Beyond the Visible Scene
In this work, we propose to look beyond the visible elements of a scene; we demonstrate that a scene is not just a collection of objects and their configuration or the labels assigned to its pixels - it is so much more.
Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection
Our features, called sketch tokens, are learned using supervised mid-level information in the form of hand drawn contours in images.
