Search Results for author:
Found 80 papers, 43 papers with code
Beyond Games: Bringing Exploration to Robots in Real-world
But most importantly, we are able to implement an exploration policy on a robot which learns to interact with objects completely from scratch just using data collected via the differentiable exploration module.
Planning to Explore via Latent Disagreement
To solve complex tasks, intelligent agents first need to explore their environments.
Learning General-Purpose Controllers via Locally Communicating Sensorimotor Modules
We observe a wide variety of drastically diverse locomotion styles across morphologies as well as centralized coordination emerging via message passing between decentralized modules purely from the reinforcement learning objective.
Evaluating Text-to-Visual Generation with Image-to-Text Generation
For instance, the widely-used CLIPScore measures the alignment between a (generated) image and text prompt, but it fails to produce reliable scores for complex prompts involving compositions of objects, attributes, and relations.
Adaptive Mobile Manipulation for Articulated Objects In the Open World
We also develop a low-cost mobile manipulation hardware platform capable of safe and autonomous online adaptation in unstructured environments with a cost of around 20, 000 USD.
Adaptive Fusion of Multi-view Remote Sensing data for Optimal Sub-field Crop Yield Prediction
The GU module learned different weights based on the country and crop-type, aligning with the variable significance of each data source to the prediction task.
PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play
To make diffusion models more useful for skill learning, we encourage robotic agents to acquire a vocabulary of skills by introducing discrete bottlenecks into the conditional behavior generation process.
Dexterous Functional Grasping
However, this task requires both a complex understanding of functional affordances as well as precise low-level control.
Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback
Our method, Diffusion-TTA, adapts pre-trained discriminative models such as image classifiers, segmenters and depth predictors, to each unlabelled example in the test set using generative feedback from a diffusion model.
DEFT: Dexterous Fine-Tuning for Real-World Hand Policies
In this paper, we investigate these challenges, especially in the case of soft, deformable objects as well as complex, relatively long-horizon tasks.
Aligning Text-to-Image Diffusion Models with Reward Backpropagation
Due to their unsupervised training, controlling their behavior in downstream tasks, such as maximizing human-perceived image quality, image-text alignment, or ethical image generation, is difficult.
Extreme Parkour with Legged Robots
In this paper, we take a similar approach to developing robot parkour on a small low-cost robot with imprecise actuation and a single front-facing depth camera for perception which is low-frequency, jittery, and prone to artifacts.
LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning
We show that LEAP Hand can be used to perform several manipulation tasks in the real world -- from visual teleoperation to learning from passive video data and sim2real.
Language Models as Black-Box Optimizers for Vision-Language Models
We highlight the advantage of conversational feedback that incorporates both positive and negative prompts, suggesting that LLMs can utilize the implicit gradient direction in textual feedback for a more efficient search.
Efficient RL via Disentangled Environment and Agent Representations
Agents that are aware of the separation between themselves and their environments can leverage this understanding to form effective representations of visual input.
Structured World Models from Human Videos
We propose an approach for robots to efficiently learn manipulation skills using only a handful of real-world interaction trajectories from many different settings.
Predicting Crop Yield With Machine Learning: An Extensive Analysis Of Input Modalities And Models On a Field and sub-field Level
We introduce a simple yet effective early fusion method for crop yield prediction that handles multiple input modalities with different temporal and spatial resolutions.
Revisiting the Role of Language Priors in Vision-Language Models
Our first observation is that they can be repurposed for discriminative tasks (such as image-text retrieval) by simply computing the match score of generating a particular text string given an image.
Ranked #45 on
Visual Reasoning
on Winoground
Affordances from Human Videos as a Versatile Representation for Robotics
Utilizing internet videos of human behavior, we train a visual affordance model that estimates where and how in the scene a human is likely to interact.
Your Diffusion Model is Secretly a Zero-Shot Classifier
Our generative approach to classification, which we call Diffusion Classifier, attains strong results on a variety of benchmarks and outperforms alternative methods of extracting knowledge from diffusion models.
Ranked #1 on
Image Classification
on ObjectNet (ImageNet classes)
Legs as Manipulator: Pushing Quadrupedal Agility Beyond Locomotion
Locomotion has seen dramatic progress for walking or running across challenging terrains.
Internet Explorer: Targeted Representation Learning on the Open Web
Modern vision models typically rely on fine-tuning general-purpose models pre-trained on large, static datasets.
ALAN: Autonomously Exploring Robotic Agents in the Real World
Robotic agents that operate autonomously in the real world need to continuously explore their environment and learn from the data collected, with minimal human supervision.
Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models
By repurposing class names as additional one-shot training samples, we achieve SOTA results with an embarrassingly simple linear classifier for vision-language adaptation.
HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration
The ability to learn from human demonstration endows robots with the ability to automate various tasks.
VideoDex: Learning Dexterity from Internet Videos
We build a learning algorithm, VideoDex, that leverages visual, action, and physical priors from human video datasets to guide robot behavior.
Legged Locomotion in Challenging Terrains using Egocentric Vision
Animals are capable of precise and agile locomotion using vision.
Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion
The standard hierarchical control pipeline for such legged manipulators is to decouple the controller into that of manipulation and locomotion.
Continual Learning with Evolving Class Ontologies
LECO requires learning classifiers in distinct time periods (TPs); each TP introduces a new ontology of "fine" labels that refines old ontologies of "coarse" labels (e. g., dog breeds that refine the previous ${\tt dog}$).
Understanding Collapse in Non-Contrastive Siamese Representation Learning
We empirically analyze these non-contrastive methods and find that SimSiam is extraordinarily sensitive to dataset and model size.
Human-to-Robot Imitation in the Wild
We approach the problem of learning by watching humans in the wild.
Adapting Rapid Motor Adaptation for Bipedal Robots
In this work, we leverage recent advances in rapid adaptation for locomotion control, and extend them to work on bipedal robots.
Topologically-Aware Deformation Fields for Single-View 3D Reconstruction
The 3D shapes are generated implicitly as deformations to a category-specific signed distance field and are learned in an unsupervised manner solely from unaligned image collections and their poses without any 3D supervision.
Test-time Adaptation with Slot-Centric Models
In our work, we find evidence that these losses are insufficient for the task of scene decomposition, without also considering architectural inductive biases.
Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube
Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem.
REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer
We interpolate between the source robot and the target robot by finding a continuous evolutionary change of robot parameters.
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
However, the plans produced naively by LLMs often cannot map precisely to admissible actions.
The CLEAR Benchmark: Continual LEArning on Real-World Imagery
The major strength of CLEAR over prior CL benchmarks is the smooth temporal evolution of visual concepts with real-world imagery, including both high-quality labeled data along with abundant unlabeled samples per time period for continual semi-supervised learning.
Functional Regularization for Reinforcement Learning via Learned Fourier Features
We propose a simple architecture for deep reinforcement learning by embedding inputs into a learned Fourier basis and show that it improves the sample efficiency of both state-based and image-based RL.
Coupling Vision and Proprioception for Navigation of Legged Robots
A safety advisor module adds sensed unexpected obstacles to the occupancy map and environment-determined speed limits to the velocity command generator.
Differentiable Spatial Planning using Transformers
We consider the problem of spatial path planning.
Interesting Object, Curious Agent: Learning Task-Agnostic Exploration
In this setup, the agent first learns to explore across many environments without any extrinsic goal in a task-agnostic manner.
Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning
We show that a single generalist policy can perform in-hand manipulation of over 100 geometrically-diverse real-world objects and generalize to new objects with unseen shape or size.
Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
An alternate but important component to consider improving is the interface of the RL algorithm with the robot.
Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots
We demonstrate that learning to minimize energy consumption plays a key role in the emergence of natural locomotion gaits at different speeds in real quadruped robots.
Discovering and Achieving Goals via World Models
How can artificial agents learn to solve many diverse tasks in complex visual environments in the absence of any supervision?
Zero-Shot Reward Specification via Grounded Natural Language
Reward signals in reinforcement learning can be expensive signals in many tasks and often require access to direct state.
How to Adapt Your Large-Scale Vision-and-Language Model
Finally, we provide an empirical analysis and recommend general recipes for efficient transfer learning of vision and language models.
Hierarchical Neural Dynamic Policies
We tackle the problem of generalization to unseen configurations for dynamic tasks in the real world while learning from high-dimensional image input.
RMA: Rapid Motor Adaptation for Legged Robots
Successful real-world deployment of legged robots would require them to adapt in real-time to unseen scenarios like changing terrains, changing payloads, wear and tear.
Unsupervised Learning of Visual 3D Keypoints for Control
Prior works show that structured latent space such as visual keypoints often outperforms unstructured representations for robotic control.
Discovering and Achieving Goals with World Models
How can an artificial agent learn to solve a wide range of tasks in a complex visual environment in the absence of external supervision?
Auto-Tuned Sim-to-Real Transfer
Policies trained in simulation often fail when transferred to the real world due to the `reality gap' where the simulator is unable to accurately capture the dynamics and visual properties of the real world.
Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image
We present Worldsheet, a method for novel view synthesis using just a single RGB image as input.
FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
A majority of methods for video frame interpolation compute bidirectional optical flow between adjacent frames of a video, followed by a suitable warping algorithm to generate the output frames.
Ranked #2 on
Video Frame Interpolation
on GoPro
Neural Dynamic Policies for End-to-End Sensorimotor Learning
We show that NDPs outperform the prior state-of-the-art in terms of either efficiency or performance across several robotic control tasks for both imitation and reinforcement learning setups.
Ranked #4 on
Meta-Learning
on MT50
Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
Learning long-term dynamics models is the key to understanding physical common sense.
Ranked #1 on
Visual Reasoning
on PHYRE-1B-Within
One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
We observe that a wide variety of drastically diverse locomotion styles across morphologies as well as centralized coordination emerges via message passing between decentralized modules purely from the reinforcement learning objective.
Locally Masked Convolution for Autoregressive Models
For tasks such as image completion, these models are unable to use much of the observed context.
Ranked #1 on
Image Generation
on MNIST
Planning to Explore via Self-Supervised World Models
Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge.
Exploring Exploration: Comparing Children with RL Agents in Unified Environments
Research in developmental psychology consistently shows that children explore the world thoroughly and efficiently and that this exploration allows them to learn.
Sparse Graphical Memory for Robust Planning
To operate effectively in the real world, agents should be able to act from high-dimensional raw sensory input such as images and achieve diverse goals across long time-horizons.
Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller
We study a generalized setup for learning from demonstration to build an agent that can manipulate novel objects in unseen scenarios by looking at only a single video of human demonstration from a third-person perspective.
Weakly-Supervised Trajectory Segmentation for Learning Reusable Skills
Instead of direct manual supervision which is tedious and prone to bias, in this work, our goal is to extract reusable skills from a collection of human demonstrations collected directly for several end-tasks.
Self-Supervised Exploration via Disagreement
In this paper, we propose a formulation for exploration inspired by the work in active learning literature.
Compositional GAN (Extended Abstract): Learning Image-Conditional Binary Composition
Generative Adversarial Networks (GANs) can produce images of surprising complexity and realism but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene.
Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity
We evaluate the performance of these dynamic and modular agents in simulated environments.
Large-Scale Study of Curiosity-Driven Learning
However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent.
Ranked #14 on
Atari Games
on Atari 2600 Montezuma's Revenge
Compositional GAN: Learning Image-Conditional Binary Composition
Generative Adversarial Networks (GANs) can produce images of remarkable complexity and realism but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene.
Learning Instance Segmentation by Interaction
The agent uses its current segmentation model to infer pixels that constitute objects and refines the segmentation model by interacting with these pixels.
Zero-Shot Visual Imitation
In our framework, the role of the expert is only to communicate the goals (i. e., what to imitate) during inference.
Investigating Human Priors for Playing Video Games
What makes humans so good at solving seemingly complex video games?
Toward Multimodal Image-to-Image Translation
Our proposed method encourages bijective consistency between the latent encoding and output modes.
Curiosity-driven Exploration by Self-supervised Prediction
In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether.
Learning Features by Watching Objects Move
Given the extensive evidence that motion plays a key role in the development of the human visual system, we hope that this straightforward approach to unsupervised learning will be more effective than cleverly designed 'pretext' tasks studied in the literature.
Context Encoders: Feature Learning by Inpainting
In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s).
Constrained Structured Regression with Convolutional Neural Networks
We present a regression framework which models the output distribution of neural networks.
Constrained Convolutional Neural Networks for Weakly Supervised Segmentation
We propose Constrained CNN (CCNN), a method which uses a novel loss function to optimize for any set of linear constraints on the output space (i. e. predicted label distribution) of a CNN.
Fully Convolutional Multi-Class Multiple Instance Learning
We propose a novel MIL formulation of multi-class semantic segmentation learning by a fully convolutional network.
Detector Discovery in the Wild: Joint Multiple Instance and Representation Learning
We develop methods for detector learning which exploit joint training over both weak and strong labels and which transfer learned perceptual representations from strongly-labeled auxiliary tasks.
