Search Results for author:
Found 19 papers, 8 papers with code
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
By training a VLM on such data, we significantly enhance its ability on both qualitative and quantitative spatial VQA.
Video Language Planning
We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web.
Towards Generalist Biomedical AI
While considerable work is needed to validate these models in real-world use cases, our results represent a milestone towards the development of generalist biomedical AI systems.
Large Language Models as General Pattern Machines
We observe that pre-trained large language models (LLMs) are capable of autoregressively completing complex token sequences -- from arbitrary ones procedurally generated by probabilistic context-free grammars (PCFG), to more rich spatial patterns found in the Abstraction and Reasoning Corpus (ARC), a general AI benchmark, prompted in the style of ASCII art.
RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning
Replicating human-like dexterity in robot hands represents one of the largest open problems in robotics.
PaLM-E: An Embodied Multimodal Language Model
Large language models excel at a wide range of complex tasks.
Ranked #2 on
Visual Question Answering (VQA)
on OK-VQA
Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents
Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models.
NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis
Expert demonstrations are a rich source of supervision for training visual robotic manipulation policies, but imitation learning methods often require either a large number of demonstrations or expensive online expert supervision to learn reactive closed-loop behaviors.
Interactive Language: Talking to Robots in Real Time
We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies).
Inner Monologue: Embodied Reasoning through Planning with Language Models
We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction.
Reinforcement Learning with Neural Radiance Fields
This paper demonstrates that learning state representations with supervision from Neural Radiance Fields (NeRFs) can improve the performance of RL compared to other learned representations or even low-dimensional, hand-engineered state information.
Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations
Our self-supervised representations are learned by observing the agent freely interacting with different parts of the environment and is queried in two different settings: (i) policy learning and (ii) object location prediction.
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Large pretrained (e. g., "foundation") models exhibit distinct capabilities depending on the domain of data they are trained on.
Ranked #21 on
Video Retrieval
on MSR-VTT-1kA
(video-to-text R@1 metric)
NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields
In particular, we demonstrate that a NeRF representation of a scene can be used to train dense object descriptors.
Implicit Behavioral Cloning
We find that across a wide range of robot policy learning scenarios, treating supervised policy learning with an implicit model generally performs better, on average, than commonly used explicit models.
XIRL: Cross-embodiment Inverse Reinforcement Learning
We investigate the visual cross-embodiment imitation setting, in which agents learn policies from videos of other agents (such as humans) demonstrating the same task, but with stark differences in their embodiments -- shape, actions, end-effector dynamics, etc.
INeRF: Inverting Neural Radiance Fields for Pose Estimation
We then show that for complex real-world scenes from the LLFF dataset, iNeRF can improve NeRF by estimating the camera poses of novel images and using these images as additional training data for NeRF.
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks
Goals cannot be as easily specified as rigid object poses, and may involve complex relative spatial relations such as "place the item inside the bag".
