Search Results for author:
Found 47 papers, 14 papers with code
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks
The IKEA Furniture Assembly Environment is one of the first benchmarks for testing and accelerating the automation of complex manipulation tasks.
Multi-view to Novel view: Synthesizing Novel Views with Self-Learned Confidence
We address the task of multi-view novel view synthesis, where we are interested in synthesizing a target image with an arbitrary camera pose from given source images.
Ranked #1 on
Novel View Synthesis
on Synthia Novel View Synthesis
Accelerating Reinforcement Learning with Learned Skill Priors
We validate our approach, SPiRL (Skill-Prior RL), on complex navigation and robotic manipulation tasks and show that learned skill priors are essential for effective skill transfer from rich datasets.
FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation
We benchmark the performance of offline RL and IL algorithms on our assembly tasks and demonstrate the need to improve such algorithms to be able to solve our tasks in the real world, providing ample opportunities for future research.
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification
When mastering a complex manipulation task, humans often decompose the task into sub-skills of their body parts, practice the sub-skills independently, and then execute the sub-skills together.
Learning to Synthesize Programs as Interpretable and Generalizable Policies
To alleviate the difficulty of learning to compose programs to induce the desired agent behavior from scratch, we propose to first learn a program embedding space that continuously parameterizes diverse behaviors in an unsupervised manner and then search over the learned program embedding space to yield a program that maximizes the return for a given task.
Generalization to New Actions in Reinforcement Learning
A fundamental trait of intelligence is the ability to achieve goals in the face of novel circumstances, such as making decisions from new action choices.
Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
Model-agnostic meta-learners aim to acquire meta-learned parameters from similar tasks to adapt to novel tasks from the same distribution with few gradient updates.
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine.
Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding
In this paper, we propose a novel policy transfer method with iterative "environment grounding", IDAPT, that alternates between (1) directly minimizing both visual and dynamics domain gaps by grounding the source environment in the target environment domains, and (2) training a policy on the grounded source environment.
Scaling simulation-to-real transfer by learning composable robot skills
In particular, we first use simulation to jointly learn a policy for a set of low-level skills, and a "skill embedding" parameterization which can be used to compose them.
Simulator Predictive Control: Using Learned Task Representations and MPC for Zero-Shot Generalization and Sequencing
We complete unseen tasks by choosing new sequences of skill latents to control the robot using MPC, where our MPC model is composed of the pre-trained skill policy executed in the simulation environment, run in parallel with the real robot.
Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation
Learning complex manipulation tasks in realistic, obstructed environments is a challenging problem due to hard exploration in the presence of obstacles and high-dimensional visual observations.
3D Interpreter Networks for Viewer-Centered Wireframe Modeling
3D-INN is trained on real images to estimate 2D keypoint heatmaps from an input image; it then predicts 3D object structure from heatmaps using knowledge learned from synthetic 3D shapes.
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
We propose an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity (e. g., "dressing") to the action (e. g., "mix yogurt") that produced it.
Single Image 3D Interpreter Network
In this work, we propose 3D INterpreter Network (3D-INN), an end-to-end framework which sequentially estimates 2D keypoint heatmaps and 3D object structure, trained on both real 2D-annotated images and synthetic 3D data.
Auto-conditioned Recurrent Mixture Density Networks for Learning Generalizable Robot Skills
Personal robots assisting humans must perform complex manipulation tasks that are typically difficult to specify in traditional motion planning pipelines, where multiple objectives must be met and the high-level context be taken into consideration.
Toward Multimodal Model-Agnostic Meta-Learning
One important limitation of such frameworks is that they seek a common initialization shared across the entire task distribution, substantially limiting the diversity of the task distributions that they are able to learn from.
Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning
Humans demonstrate remarkable abilities to predict physical events in dynamic scenes, and to infer the physical properties of objects from static images.
Demo2Vec: Reasoning Object Affordances From Online Videos
Watching expert demonstrations is an important way for humans and robots to reason about affordances of unseen objects.
Ranked #2 on
Video-to-image Affordance Grounding
on OPRA (28x28)
Learning and Using the Arrow of Time
We seek to understand the arrow of time in videos -- what makes videos look like they are playing forwards or backwards?
Ranked #50 on
Self-Supervised Action Recognition
on UCF101
Self-Supervised Action Recognition
Temporal Action Localization
+1
Composing Complex Skills by Learning Transition Policies with Proximity Reward Induction
Intelligent creatures acquire complex skills by exploiting previously learned skills and learning to transition between them.
Learning from Noisy Demonstration Sets via Meta-Learned Suitability Assessor
A noisy and diverse demonstration set may hinder the performances of an agent aiming to acquire certain skills via imitation learning.
Universal Agent for Disentangling Environments and Tasks
Recent state-of-the-art reinforcement learning algorithms are trained under the goal of excelling in one specific task.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection
Our features, called sketch tokens, are learned using supervised mid-level information in the form of hand drawn contours in images.
Looking Beyond the Visible Scene
In this work, we propose to look beyond the visible elements of a scene; we demonstrate that a scene is not just a collection of objects and their configuration or the labels assigned to its pixels - it is so much more.
Discovering States and Transformations in Image Collections
Our system works by generalizing across object classes: states and transformations learned on one set of objects are used to interpret the image collection for an entirely new object class.
Knowledge Acquisition for Visual Question Answering via Iterative Querying
Humans possess an extraordinary ability to learn new skills and new knowledge for problem solving.
To Follow or not to Follow: Selective Imitation Learning from Observations
Learning from demonstrations is a useful way to transfer a skill from one agent to another.
Program Guided Agent
Developing agents that can learn to follow natural language instructions has been an emerging research direction.
Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments
In contrast, motion planners use explicit models of the agent and environment to plan collision-free paths to faraway goals, but suffer from inaccurate models in tasks that require contacts with the environment.
Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning
Active learning is widely used to reduce labeling effort and training time by repeatedly querying only the most beneficial samples from unlabeled data.
Demonstration-Guided Reinforcement Learning with Learned Skills
Prior approaches for demonstration-guided RL treat every new task as an independent learning problem and attempt to follow the provided demonstrations step-by-step, akin to a human trying to imitate a completely unseen behavior by following the demonstrator's exact muscle movements.
Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
However, these approaches require larger state distributions to be covered as more policies are sequenced, and thus are limited to short skill sequences.
Generalizable Imitation Learning from Observation via Inferring Goal Proximity
Task progress is intuitive and readily available task information that can guide an agent closer to the desired goal.
Model-Agnostic Meta-Learning for Multimodal Task Distributions
In this paper, we augment MAML with the capability to identify tasks sampled from a multimodal task distribution and adapt quickly through gradient updates.
Generalizing Reinforcement Learning to Unseen Actions
Hence, we propose a framework to enable generalization over both these aspects: understanding an action’s functionality, and using actions to solve tasks through reinforcement learning.
Keyframing the Future: Discovering Temporal Hierarchy with Keyframe-Inpainter Prediction
To flexibly and efficiently reason about temporal sequences, abstract representations that compactly represent the important information in the sequence are needed.
Task-Induced Representation Learning
We investigate the effectiveness of unsupervised and task-induced representation learning approaches on four visually complex environments, from Distracting DMControl to the CARLA driving simulator.
Skill-based Model-based Reinforcement Learning
From this intuition, we propose a Skill-based Model-based RL framework (SkiMo) that enables planning in the skill space using a skill dynamics model, which directly predicts the skill outcomes, rather than predicting all small details in the intermediate states, step by step.
Model-based Reinforcement Learning
reinforcement-learning
+1
PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection
Large-scale data is an essential component of machine learning as demonstrated in recent advances in natural language processing and computer vision research.
Cross-Domain Transfer via Semantic Skill Imitation
We propose an approach for semantic imitation, which uses demonstrations from a source domain, e. g. human videos, to accelerate reinforcement learning (RL) in a different target domain, e. g. a robotic manipulator in a simulated kitchen.
Efficient Multi-Task Reinforcement Learning via Selective Behavior Sharing
The ability to leverage shared behaviors between tasks is critical for sample-efficient multi-task reinforcement learning (MTRL).
Hierarchical Neural Program Synthesis
Then, we train a high-level module to comprehend the task specification (e. g., input/output pairs or demonstrations) from long programs and produce a sequence of task embeddings, which are then decoded by the program decoder and composed to yield the synthesized program.
SPRINT: Scalable Policy Pre-Training via Language Instruction Relabeling
Pre-training robot policies with a rich set of skills can substantially accelerate the learning of downstream tasks.
Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidance
Instead, our approach BOSS (BOotStrapping your own Skills) learns to accomplish new tasks by performing "skill bootstrapping," where an agent with a set of primitive skills interacts with the environment to practice new skills without receiving reward feedback for tasks outside of the initial skill set.
LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers
We propose a framework that leverages foundation models as teachers, guiding a reinforcement learning agent to acquire semantically meaningful behavior without human feedback.
