Search Results for author:
Found 64 papers, 23 papers with code
Compositional Foundation Models for Hierarchical Planning
We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model.
Compositional Diffusion-Based Continuous Constraint Solvers
This paper introduces an approach for learning to solve continuous constraint satisfaction problems (CCSP) in robotic reasoning and planning.
3D-LLM: Injecting the 3D World into Large Language Models
Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs.
Building Cooperative Embodied Agents Modularly with Large Language Models
Large Language Models (LLMs) have demonstrated impressive planning abilities in single-agent embodied tasks across various domains.
Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models
Text-to-image generative models have enabled high-resolution image synthesis across different domains, but require users to specify the content they wish to generate.
Probabilistic Adaptation of Text-to-Video Models
Large text-to-video models trained on internet-scale data have demonstrated exceptional capabilities in generating high-fidelity videos from arbitrary textual descriptions.
FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow
SE(3) camera pose estimation is then performed via a weighted least-squares fit to the scene flow field.
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks.
Training Diffusion Models with Reinforcement Learning
Diffusion models are a class of flexible generative models trained with an approximation to the log-likelihood objective.
Learning to Render Novel Views from Wide-Baseline Stereo Pairs
We conduct extensive comparisons on held-out test scenes across two real-world datasets, significantly outperforming prior work on novel view synthesis from sparse image observations and achieving multi-view-consistent novel view synthesis.
Planning with Sequence Models through Iterative Energy Minimization
In this paper, we suggest an approach towards integrating planning with sequence models based on the idea of iterative energy minimization, and illustrate how such a procedure leads to improved RL performance across different tasks.
3D Concept Learning and Reasoning from Multi-View Images
We suggest that a principled approach for 3D reasoning from multi-view images should be to infer a compact 3D representation of the world from the multi-view images, which is further grounded on open-vocabulary semantic concepts, and then to execute reasoning on these 3D representations.
NeuSE: Neural SE(3)-Equivariant Embedding for Consistent Spatial Understanding with Objects
We present NeuSE, a novel Neural SE(3)-Equivariant Embedding for objects, and illustrate how it supports object SLAM for consistent spatial understanding with long-term scene changes.
Foundation Models for Decision Making: Problems, Methods, and Opportunities
In response to these developments, new paradigms are emerging for training foundation models to interact with other agents and perform long-term reasoning.
Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance.
Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation
A robot operating in a household environment will see a wide range of unique and unfamiliar objects.
Learning Universal Policies via Text-Guided Video Generation
The proposed policy-as-video formulation can further represent environments with different state and action spaces in a unified space of images, which, for example, enables learning and generalization across a variety of robot manipulation tasks.
Is Conditional Generative Modeling all you need for Decision-Making?
We further demonstrate the advantages of modeling policies as conditional diffusion models by considering two other conditioning variables: constraints and skills.
SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields
This formalism is implemented in three steps: assigning a consistent local coordinate frame to the task-relevant object parts, determining the location and orientation of this coordinate frame on unseen object instances, and executing an action that brings these frames into the desired alignment.
Self-conditioned Embedding Diffusion for Text Generation
Can continuous diffusion models bring the same performance breakthrough on natural language they did for image generation?
StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects
StructDiffusion even improves the success rate of assembling physically-valid structures out of unseen objects by on average 16% over an existing multi-modal transformer model trained on specific structures.
Composing Ensembles of Pre-trained Models via Iterative Consensus
Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e. g. improving accuracy on grade school math problems by 7. 5%, without requiring any model finetuning.
Ranked #1 on
Video Question Answering
on ActivityNet-QA
(Vocabulary Size metric)
Robust Change Detection Based on Neural Descriptor Fields
The ability to reason about changes in the environment is crucial for robots operating over extended periods of time.
Neural Groundplans: Persistent Neural Scene Representations from a Single Image
We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene.
3D Concept Grounding on Neural Fields
Experimental results show that our proposed framework outperforms unsupervised/language-mediated segmentation models on semantic and instance segmentation tasks, as well as outperforms existing models on the challenging 3D aware visual reasoning tasks.
Learning Iterative Reasoning through Energy Minimization
Finally, we illustrate that our approach can recursively solve algorithmic problems requiring nested reasoning
Compositional Visual Generation with Composable Diffusion Models
Large text-guided diffusion models, such as DALLE-2, are able to generate stunning photorealistic images given natural language descriptions.
Planning with Diffusion for Flexible Behavior Synthesis
Model-based reinforcement learning methods often use learning only for the purpose of estimating an approximate dynamics model, offloading the rest of the decision-making work to classical trajectory optimizers.
Streaming Inference for Infinite Non-Stationary Clustering
Learning from a continuous stream of non-stationary data in an unsupervised manner is arguably one of the most common and most challenging settings facing intelligent agents.
Learning Neural Acoustic Fields
By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds.
Kubric: A scalable dataset generator
Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details.
Pre-Trained Language Models for Interactive Decision-Making
Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation
Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors.
Learning to Compose Visual Relations
The visual world around us can be described as a structured set of objects and their associated relations.
Learning Signal-Agnostic Manifolds of Neural Fields
We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains in a modality-independent manner.
Unsupervised Learning of Compositional Energy Concepts
In this work, we propose COMET, which discovers and represents concepts as separate energy functions, enabling us to represent both global concepts as well as objects under a unified framework.
The Neural MMO Platform for Massively Multiagent Research
Neural MMO is a computationally accessible research platform that combines large agent populations, long time horizons, open-ended tasks, and modular game systems.
Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Our task poses unique challenges as a system does not know what types of human-object interactions are present in a video or the actual spatiotemporal location of the human and the object.
Human-Object Interaction Detection
Weakly-supervised Learning
Language Model Pre-training Improves Generalization in Policy Learning
Additional experiments explore the role of language-based encodings in these results; we find that it is possible to train a simple adapter layer that maps from observations and action histories to LM embeddings, and thus that language modeling provides an effective initializer even for tasks with no language as input or output.
Curious Representation Learning for Embodied Intelligence
Instead, it must explore its environment to acquire the data it will learn from.
3D Shape Generation and Completion through Point-Voxel Diffusion
We propose a novel approach for probabilistic generative modeling of 3D shapes.
Unsupervised Discovery of 3D Physical Objects
We study the problem of unsupervised physical object discovery.
Neural Radiance Flow for 4D View Synthesis and Video Processing
We present a method, Neural Radiance Flow (NeRFlow), to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images.
Improved Contrastive Divergence Training of Energy Based Models
Contrastive divergence is a popular method of training energy-based models, but is known to have difficulties with training stability.
Compositional Visual Generation with Energy Based Models
A vital aspect of human intelligence is the ability to compose increasingly complex concepts out of simpler ideas, enabling both rapid learning and adaptation of knowledge.
Energy-Based Models for Continual Learning
We motivate Energy-Based Models (EBMs) as a promising model class for continual learning problems.
A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
We present a framework for solving long-horizon planning problems involving manipulation of rigid objects that operates directly from a point-cloud observation, i. e. without prior object models.
Learning Object-Based State Estimators for Household Robots
The robot may be called upon later to retrieve objects and will need a long-term object-based memory in order to know how to find them.
Learning Online Data Association
When an agent interacts with a complex environment, it receives a stream of percepts in which it may detect entities, such as objects or people.
Unsupervised Discovery of 3D Physical Objects from Video
We study the problem of unsupervised physical object discovery.
Energy-based models for atomic-resolution protein conformations
We propose an energy-based model (EBM) of protein conformations that operates at atomic scale.
Compositional Visual Generation and Inference with Energy Based Models
A vital aspect of human intelligence is the ability to compose increasingly complex concepts out of simpler ideas, enabling both rapid learning and adaptation of knowledge.
Neural MMO v1.3: A Massively Multiagent Game Environment for Training and Evaluating Neural Networks
We present Neural MMO, a massively multiagent game environment inspired by MMOs and discuss our progress on two more general challenges in multiagent systems engineering for AI research: distributed infrastructure and game IO.
Observational Overfitting in Reinforcement Learning
A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP).
Implicit Generation and Modeling with Energy Based Models
Energy based models (EBMs) are appealing due to their generality and simplicity in likelihood modeling, but have been traditionally difficult to train.
Ranked #3 on
Image Generation
on Stacked MNIST
Learning Good Policies By Learning Good Perceptual Models
Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years.
Model Based Planning with Energy Based Models
We provide an online algorithm to train EBMs while interacting with the environment, and show that EBMs allow for significantly better online learning than corresponding feed-forward networks.
An Empirical Study on Hyperparameters and their Interdependence for RL Generalization
Recent results in Reinforcement Learning (RL) have shown that agents with limited training environments are susceptible to a large amount of overfitting across many domains.
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
While model-based deep reinforcement learning (RL) holds great promise for sample efficiency and generalization, learning an accurate dynamics model is often challenging and requires substantial interaction with the environment.
Neural MMO: A massively multiplayer game environment for intelligent agents
We demonstrate how this platform can be used to study behavior and learning in large populations of neural agents.
Implicit Generation and Generalization in Energy-Based Models
Energy based models (EBMs) are appealing due to their generality and simplicity in likelihood modeling, but have been traditionally difficult to train.
Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents
The emergence of complex life on Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for finite resources.
Learning to Exploit Stability for 3D Scene Parsing
We first show that applying physics supervision to an existing scene understanding model increases performance, produces more stable predictions, and allows training to an equivalent performance level with fewer annotated training examples.
Learning Physics Priors for Deep Reinforcement Learing
While model-based deep reinforcement learning (RL) holds great promise for sample efficiency and generalization, learning an accurate dynamics model is challenging and often requires substantial interactions with the environment.
