Search Results for author:
Found 30 papers, 10 papers with code
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers
Given a video demonstration of a manipulation task and current visual observations, Vid2Robot directly produces robot actions.
RT-H: Action Hierarchies Using Language
Predicting these language motions as an intermediate step between tasks and actions forces the policy to learn the shared structure of low-level motions across seemingly disparate tasks.
AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents
We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences.
Video Language Planning
We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data.
Robotic Table Tennis: A Case Study into a High Speed Learning System
We present a deep-dive into a real-world robotic learning system that, in previous work, was shown to be capable of hundreds of table tennis rallies with a human and has the ability to precisely return the ball to desired targets.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web.
PaLM-E: An Embodied Multimodal Language Model
Large language models excel at a wide range of complex tasks.
Ranked #2 on
Visual Question Answering (VQA)
on OK-VQA
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
To accomplish this, we introduce Data-driven Instruction Augmentation for Language-conditioned control (DIAL): we utilize semi-supervised language labels leveraging the semantic understanding of CLIP to propagate knowledge onto large datasets of unlabelled demonstration data and then train language-conditioned policies on the augmented datasets.
Inner Monologue: Embodied Reasoning through Planning with Language Models
We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction.
Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations
Our self-supervised representations are learned by observing the agent freely interacting with different parts of the environment and is queried in two different settings: (i) policy learning and (ii) object location prediction.
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment.
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
On semi-supervised learning benchmarks we improve performance significantly when only 1% ImageNet labels are available, from 53. 8% to 56. 5%.
Ranked #1 on
Image Classification
on PASCAL VOC 2007
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
This task space can be quite general and abstract; its only requirements are to be sampleable and to well-cover the space of useful tasks.
Counting Out Time: Class Agnostic Video Repetition Counting in the Wild
We present an approach for estimating the period with which an action is repeated in a video.
Learning to Play by Imitating Humans
Acquiring multiple skills has commonly involved collecting a large number of expert demonstrations per task or engineering custom reward functions.
Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos
We demonstrate the use of this representation to imitate surgical suturing motions from publicly available videos of the JIGSAWS dataset.
Language Conditioned Imitation Learning over Unstructured Data
Prior work in imitation learning typically requires each task be specified with a task id or goal image -- something that is often impractical in open-world environments.
Online Object Representations with Contrastive Learning
We propose a self-supervised approach for learning representations of objects from monocular videos and demonstrate it is particularly useful in situated settings such as robotics.
Temporal Cycle-Consistency Learning
We introduce a self-supervised representation learning method based on the task of temporal alignment between videos.
Ranked #1 on
Video Alignment
on UPenn Action
Wasserstein Dependency Measure for Representation Learning
Mutual information maximization has emerged as a powerful learning objective for unsupervised representation learning obtaining state-of-the-art performance in applications such as object recognition, speech recognition, and reinforcement learning.
Learning Latent Plans from Play
Learning from play (LfP) offers three main advantages: 1) It is cheap.
Robotics
Learning Actionable Representations from Visual Observations
In this work we explore a new approach for robots to teach themselves about the world simply by observing it.
Time-Contrastive Networks: Self-Supervised Learning from Video
While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human.
Ranked #3 on
Video Alignment
on UPenn Action
Unsupervised Perceptual Rewards for Imitation Learning
We present a method that is able to identify key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps.
Attention for Fine-Grained Categorization
This paper presents experiments extending the work of Ba et al. (2014) on recurrent neural models for attention into less constrained visual environments, specifically fine-grained categorization on the Stanford Dogs data set.
Going Deeper with Convolutions
We propose a deep convolutional neural network architecture codenamed "Inception", which was responsible for setting the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC 2014).
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
This integrated framework is the winner of the localization task of the ImageNet Large Scale Visual Recognition Challenge 2013 (ILSVRC2013) and obtained very competitive results for the detection and classifications tasks.
Pedestrian Detection with Unsupervised Multi-Stage Feature Learning
Pedestrian detection is a problem of considerable practical interest.
Convolutional Neural Networks Applied to House Numbers Digit Classification
We classify digits of real-world house numbers using convolutional neural networks (ConvNets).
Learning Convolutional Feature Hierarchies for Visual Recognition
We propose an unsupervised method for learning multi-stage hierarchies of sparse convolutional features.
