Search Results for author:
Found 56 papers, 23 papers with code
Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation
We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation: interacting with unseen objects in novel scenes without test-time adaptation.
G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis
We propose G-HOP, a denoising diffusion based generative prior for hand-object interactions that allows modeling both the 3D object and a human hand, conditioned on the object category.
MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation
We present MVD-Fusion: a method for single-view 3D inference via generative modeling of multi-view-consistent RGB-D images.
Cameras as Rays: Pose Estimation via Ray Diffusion
Estimating camera poses is a fundamental task for 3D reconstruction and remains challenging given sparsely sampled views (<10).
UpFusion: Novel View Diffusion from Unposed Sparse View Observations
We propose UpFusion, a system that can perform novel view synthesis and infer 3D representations for an object given a sparse set of reference images without corresponding pose information.
Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans
We pursue the goal of developing robots that can interact zero-shot with generic unseen objects via a diverse repertoire of manipulation skills and show how passive human videos can serve as a rich source of data for learning such generalist robots.
Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips
We tackle the task of reconstructing hand-object interactions from short video clips.
RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking
The grand aim of having a single robot that can manipulate arbitrary objects in diverse settings is at odds with the paucity of robotics datasets.
Visual Affordance Prediction for Guiding Robot Exploration
Motivated by the intuitive understanding humans have about the space of possible interactions, and the ease with which they can generalize this understanding to previously unseen scenes, we develop an approach for learning visual affordances for guiding robot exploration.
RelPose++: Recovering 6D Poses from Sparse-view Observations
We address the task of estimating 6D camera poses from sparse-view image sets (2-8 images).
Analogy-Forming Transformers for Few-Shot 3D Parsing
We present Analogical Networks, a model that encodes domain knowledge explicitly, in a collection of structured labelled 3D scenes, in addition to implicitly, as model parameters, and segments 3D object scenes with analogical reasoning: instead of mapping a scene to part segments directly, our model first retrieves related scenes from memory and their corresponding part structures, and then predicts analogous part structures for the input scene, via an end-to-end learnable modulation mechanism.
Mesh2Tex: Generating Mesh Textures from Image Queries
The learned texture manifold enables effective navigation to generate an object texture for a given 3D object geometry that matches to an input RGB image, which maintains robustness even under challenging real-world scenarios where the mesh geometry approximates an inexact match to the underlying geometry in the RGB image.
Affordance Diffusion: Synthesizing Hand-Object Interactions
In contrast, in this work we focus on synthesizing complex interactions (ie, an articulated hand) with a given object.
Zero-Shot Robot Manipulation from Passive Human Videos
Can we learn robot manipulation for everyday tasks, only by watching videos of humans doing arbitrary tasks in different unstructured settings?
Geometry-biased Transformers for Novel View Synthesis
While this representation yields (coarsely) accurate images corresponding to novel viewpoints, the lack of geometric reasoning limits the quality of these outputs.
SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction
We propose SparseFusion, a sparse view 3D reconstruction approach that unifies recent advances in neural rendering and probabilistic image generation.
Monocular Dynamic View Synthesis: A Reality Check
We study the recent progress on dynamic view synthesis (DVS) from monocular video.
RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild
We describe a data-driven method for inferring the camera viewpoints given multiple images of an arbitrary object.
What's in your hands? 3D Reconstruction of Generic Objects in Hands
Our work aims to reconstruct hand-held objects given a single RGB image.
Pre-train, Self-train, Distill: A simple recipe for Supersizing 3D Reconstruction
Our final 3D reconstruction model is also capable of zero-shot inference on images from unseen object categories and we empirically show that increasing the number of training categories improves the reconstruction quality.
AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation
This enables us to represent distributions over 3D shapes conditioned on information from an arbitrary set of spatially anchored query locations and thus perform shape completion in such arbitrary settings (e. g., generating a complete chair given only a view of the back leg).
A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation
Specifying tasks with videos is a powerful technique towards acquiring novel and general robot skills.
No RL, No Simulation: Learning to Navigate without Navigating
Most prior methods for learning navigation policies require access to simulation environments, as they need online policy interaction and rely on ground-truth maps for rewards.
NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild
NeRS learns a neural shape representation of a closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions.
PixelTransformer: Sample Conditioned Signal Generation
We propose a generative model that can infer a distribution for the underlying spatial signal conditioned on sparse samples e. g. plausible images given a few observed pixels.
Shelf-Supervised Mesh Prediction in the Wild
We first infer a volumetric representation in a canonical frame, along with the camera pose.
Where2Act: From Pixels to Actions for Articulated 3D Objects
One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment.
Visual Imitation Made Easy
We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector.
Object-Centric Multi-View Aggregation
We present an approach for aggregating a sparse set of views of an object in order to compute a semi-implicit 3D representation in the form of a volumetric feature grid.
Implicit Mesh Reconstruction from Unannotated Image Collections
We present an approach to infer the 3D shape, texture, and camera pose for an object from a single RGB image, using only category-level image collections with foreground masks as supervision.
See, Hear, Explore: Curiosity via Audio-Visual Association
Exploration is one of the core challenges in reinforcement learning.
Articulation-aware Canonical Surface Mapping
We tackle the tasks of: 1) predicting a Canonical Surface Mapping (CSM) that indicates the mapping from 2D pixels to corresponding points on a canonical template shape, and 2) inferring the articulation and pose of the template corresponding to the input image.
Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects
Our quantitative and qualitative results show that (a) we can predict meaningful forces from videos whose effects lead to accurate imitation of the motions observed, (b) by jointly optimizing for contact point and force prediction, we can improve the performance on both tasks in comparison to independent training, and (c) we can learn a representation from this model that generalizes to novel objects using few shot examples.
Intrinsic Motivation for Encouraging Synergistic Behavior
Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own.
Discovering Motor Programs by Recomposing Demonstrations
In this paper, we present an approach to learn recomposable motor primitives across large-scale and diverse manipulation demonstrations.
Object-centric Forward Modeling for Model Predictive Control
We present an approach to learn an object-centric forward model, and show that this allows us to plan for sequences of actions to achieve distant desired goals.
Efficient Bimanual Manipulation Using Learned Task Schemas
Our insight is that for many tasks, the learning process can be decomposed into learning a state-independent task schema (a sequence of skills to execute) and a policy to choose the parameterizations of the skills in a state-dependent manner.
Compositional Video Prediction
We present an approach for pixel-level future prediction given an input image of a scene.
Canonical Surface Mapping via Geometric Cycle Consistency
We explore the task of Canonical Surface Mapping (CSM).
3D-RelNet: Joint Object and Relational Network for 3D Prediction
We propose an approach to predict the 3D shape and pose for the objects present in a scene.
Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency
We demonstrate our ability to learn MVS without 3D supervision using a real dataset, and show that each component of our proposed robust loss results in a significant improvement.
Layer-structured 3D Scene Inference via View Synthesis
We present an approach to infer a layer-structured 3D representation of a scene from a single input image.
Learning Category-Specific Mesh Reconstruction from Image Collections
The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation.
Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction
We present a framework for learning single-view shape and pose prediction without using direct supervision for either.
Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene
The goal of this paper is to take a single 2D image of a scene and recover the 3D structure in terms of a small set of factors: a layout representing the enclosing surfaces as well as a set of objects represented in terms of shape and pose.
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database.
Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency
We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view.
Hierarchical Surface Prediction for 3D Object Reconstruction
A major limitation of such approaches is that they only predict a coarse resolution voxel grid, which does not capture the surface of the objects well.
Learning Shape Abstractions by Assembling Volumetric Primitives
We present a learning framework for abstracting complex shapes by learning to assemble objects using 3D volumetric primitives.
View Synthesis by Appearance Flow
We address the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints.
Shape and Symmetry Induction for 3D Objects
Actions as simple as grasping an object or navigating around it require a rich understanding of that object's 3D shape from a given viewpoint.
Amodal Completion and Size Constancy in Natural Scenes
We consider the problem of enriching current object detection systems with veridical object sizes and relative depth estimates from a single image.
Pose Induction for Novel Object Categories
We address the task of predicting pose for objects of unannotated object categories from a small seed set of annotated object classes.
Virtual View Networks for Object Reconstruction
All that structure from motion algorithms "see" are sets of 2D points.
Category-Specific Object Reconstruction from a Single Image
Object reconstruction from a single image -- in the wild -- is a problem where we can make progress and get meaningful results today.
Viewpoints and Keypoints
We characterize the problem of pose estimation for rigid objects in terms of determining viewpoint to explain coarse pose and keypoint prediction to capture the finer details.
Ranked #3 on
Keypoint Detection
on Pascal3D+
