Search Results for author:
Found 164 papers, 92 papers with code
When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?
Visual reinforcement learning is a promising approach that directly trains policies from visual observations, although it faces challenges in sample efficiency and computational costs.
Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model
Recent advancements in robot learning have used imitation learning with large models and extensive demonstrations to develop effective policies.
ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks
To this end, we present MS-HAB, a holistic benchmark for low-level manipulation and in-home object rearrangement.
3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation
Multi-view image diffusion models have significantly advanced open-domain 3D object generation.
Reward-free World Models for Online Imitation Learning
By employing a learned latent dynamics model and planning for control, our approach consistently achieves stable, expert-level performance in tasks with high-dimensional observation or action spaces and intricate dynamics.
ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI
We introduce and open source ManiSkill3, the fastest state-visual GPU parallelized robotics simulator with contact-rich physics targeting generalizable manipulation.
ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Images
To advance the state of the art in the creation of 3D foundation models, this paper introduces the ConDense framework for 3D pre-training utilizing existing pre-trained 2D networks and large-scale multi-view datasets.
SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Views
The diffusion model is trained to jointly predict surrogate representations for camera poses and multi-view images of the object under known poses, integrating all information from the input sparse views.
MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model
The input normal maps can be predicted by 2D diffusion models, significantly aiding in the guidance and refinement of the geometry's learning.
Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-Rank Decomposition
Existing PFL methods primarily adopt a parameter partitioning approach, where the parameters of a model are designated as one of two types: parameters shared with other clients to extract general knowledge and parameters retained locally to learn client-specific knowledge.
Point-SAM: Promptable 3D Segmentation Model for Point Clouds
The development of 2D foundation models for image segmentation has been significantly advanced by the Segment Anything Model (SAM).
Estimating before Debiasing: A Bayesian Approach to Detaching Prior Bias in Federated Semi-Supervised Learning
Federated Semi-Supervised Learning (FSSL) leverages both labeled and unlabeled data on clients to collaboratively train a model. In FSSL, the heterogeneous data can introduce prediction bias into the model, causing the model's prediction to skew towards some certain classes.
Hierarchical World Models as Visual Whole-Body Humanoid Controllers
Whole-body control for humanoids is challenging due to the high-dimensional nature of the problem, coupled with the inherent instability of a bipedal morphology.
ElastoGen: 4D Generative Elastodynamics
We present ElastoGen, a knowledge-driven AI model that generates physically accurate 4D elastodynamics.
Evaluating Real-World Robot Manipulation Policies in Simulation
We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups.
Reverse Forward Curriculum Learning for Extreme Sample and Demonstration Efficiency in Reinforcement Learning
A forward curriculum is then used to accelerate the training of the initial policy to perform well on the full initial state distribution of the task and improve demonstration and sample efficiency.
A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose
We also introduce a novel notion of an expected surface in Gaussian splatting, which is critical to our optimization.
Part-Guided 3D RL for Sim2Real Articulated Object Manipulation
Manipulating unseen articulated objects through visual feedback is a critical but challenging task for real robots.
DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks
The success of many RL techniques heavily relies on human-engineered dense rewards, which typically demand substantial domain expertise and extensive trial and error.
Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos
We introduce the Gaussian-Mesh Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh-guided densification and pruning on the deformed Gaussians.
MeshLRM: Large Reconstruction Model for High-Quality Mesh
This allows for end-to-end mesh reconstruction by fine-tuning a pre-trained NeRF LRM with mesh rendering.
AdaDemo: Data-Efficient Demonstration Expansion for Generalist Robotic Agent
Encouraged by the remarkable achievements of language and vision foundation models, developing generalist robotic agents through imitation learning, using large demonstration datasets, has become a prominent area of interest in robot learning.
Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D
In recent years, there has been an explosion of 2D vision models for numerous tasks such as semantic segmentation, style transfer or scene editing, enabled by large-scale 2D image datasets.
Generic 3D Diffusion Adapter Using Controlled Multi-View Editing
Open-domain 3D object synthesis has been lagging behind image synthesis due to limited data and higher computational complexity.
CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation
We introduce CyberDemo, a novel approach to robotic imitation learning that leverages simulated human demonstrations for real-world tasks.
Constrained Online Two-stage Stochastic Optimization: Algorithm with (and without) Predictions
When the model parameters are drawn from unknown non-stationary distributions and we are given machine-learned predictions of the distributions, we develop a new algorithm from our framework with a regret $O(W_T+\sqrt{T})$, where $W_T$ measures the total inaccuracy of the machine-learned predictions.
ZeroRF: Fast Sparse View 360deg Reconstruction with Zero Pretraining
We present ZeroRF a novel per-scene optimization method addressing the challenge of sparse view 360deg reconstruction in neural field representations.
ZeroRF: Fast Sparse View 360° Reconstruction with Zero Pretraining
We present ZeroRF, a novel per-scene optimization method addressing the challenge of sparse view 360{\deg} reconstruction in neural field representations.
DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
Combining gradient-based trajectory optimization with differentiable physics simulation is an efficient technique for solving soft-body manipulation problems.
Robo360: A 3D Omnispective Multi-Material Robotic Manipulation Dataset
Building robots that can automate labor-intensive tasks has long been the core motivation behind the advancements in computer vision and the robotics community.
PartSLIP++: Enhancing Low-Shot 3D Part Segmentation via Multi-View Instance Segmentation and Maximum Likelihood Estimation
Open-world 3D part segmentation is pivotal in diverse applications such as robotics and AR/VR.
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion
Recent advancements in open-world 3D object generation have been remarkable, with image-to-3D methods offering superior fine-grained control over their text-to-3D counterparts.
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Generalization remains one of the most important desiderata for robust robot learning systems.
Unleashing the Creative Mind: Language Model As Hierarchical Policy For Improved Exploration on Challenging Problem Solving
Large Language Models (LLMs) have achieved tremendous progress, yet they still often struggle with challenging reasoning problems.
TD-MPC2: Scalable, Robust World Models for Continuous Control
TD-MPC is a model-based reinforcement learning (RL) algorithm that performs local trajectory optimization in the latent space of a learned implicit (decoder-free) world model.
Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model
We report Zero123++, an image-conditioned diffusion model for generating 3D-consistent multi-view images from a single input view.
FG-NeRF: Flow-GAN based Probabilistic Neural Radiance Field for Independence-Assumption-Free Uncertainty Estimation
Existing works rely on the independence assumption of points in the radiance field or the pixels in input views to obtain tractable forms of the probability density function.
OpenIllumination: A Multi-Illumination Dataset for Inverse Rendering Evaluation on Real Objects
We introduce OpenIllumination, a real-world dataset containing over 108K images of 64 objects with diverse materials, captured under 72 camera views and a large number of different illuminations.
Strivec: Sparse Tri-Vector Radiance Fields
In contrast to TensoRF which uses a global tensor and focuses on their vector-matrix decomposition, we propose to utilize a cloud of local tensors and apply the classic CANDECOMP/PARAFAC (CP) decomposition to factorize each tensor into triple vectors that express local feature distributions along spatial axes and compactly encode a local neural field.
Reparameterized Policy Learning for Multimodal Trajectory Optimization
We then present a practical model-based RL method, called Reparameterized Policy Gradient (RPG), which leverages the multimodal policy parameterization and learned world model to achieve strong exploration capabilities and high data efficiency.
3Deformer: A Common Framework for Image-Guided Mesh Deformation
Unlike these studies, our 3Deformer is a non-training and common framework, which only requires supervision of readily-available semantic images, and is compatible with editing various objects unlimited by datasets.
AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System
For real-world experiments, AnyTeleop can outperform a previous system that was designed for a specific robot hardware with a higher success rate, using the same robot.
Distilling Large Vision-Language Model with Out-of-Distribution Generalizability
Model distillation, the process of creating smaller, faster models that maintain the performance of larger models, is a promising direction towards the solution.
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization
Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world.
On the Efficacy of 3D Point Cloud Reinforcement Learning
Recent studies on visual reinforcement learning (visual RL) have explored the use of 3D visual representations.
Deductive Verification of Chain-of-Thought Reasoning
In light of this, we propose to decompose a reasoning verification process into a series of step-by-step subprocesses, each only receiving their necessary context and premises.
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models
Image ad understanding is a crucial task with wide real-world applications.
NeuManifold: Neural Watertight Manifold Reconstruction with Efficient and High-Quality Rendering Support
We present a method for generating high-quality watertight manifold meshes from multi-view input images.
OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding
Due to their alignment with CLIP embeddings, our learned shape representations can also be integrated with off-the-shelf CLIP-based models for various applications, such as point cloud captioning and point cloud-conditioned image generation.
Ranked #5 on
Zero-shot 3D Point Cloud Classification
on OmniObject3D (Pretrained on ShapeNet)
(using extra training data)
EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration
Hand-eye calibration is a critical task in robotics, as it directly affects the efficacy of critical operations such as manipulation and grasping.
TensoIR: Tensorial Inverse Rendering
We propose TensoIR, a novel inverse rendering approach based on tensor factorization and neural fields.
Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images.
Chain-of-Thought Predictive Control
By grouping temporarily close and functionally similar actions into subskill-level demo segments, the observations at the segment boundaries constitute a chain of planning steps for the task, which we refer to as the chain-of-thought (CoT).
DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics
Reinforcement learning approaches for dexterous rigid object manipulation would struggle in this setting due to the complexity of physics interaction with deformable objects.
Boosting Reinforcement Learning and Planning with Demonstrations: A Survey
Although reinforcement learning has seen tremendous success recently, this kind of trial-and-error learning can be impractical or inefficient in complex environments.
MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Field
Under the Lagrangian view, we parameterize the scene motion by tracking the trajectory of particles on objects.
Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervision
We address efficient and structure-aware 3D scene representation from images.
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
Generalizable manipulation skills, which can be composed to tackle long-horizon and complex daily chores, are one of the cornerstones of Embodied AI.
Factor Fields: A Unified Framework for Neural Fields and Beyond
Our experiments show that DiF leads to improvements in approximation quality, compactness, and training time when compared to previous fast reconstruction methods.
On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline
In this paper, we examine the effectiveness of pre-training for visuo-motor control tasks.
MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
We identify key ingredients for leveraging demonstrations in model learning -- policy pretraining, targeted exploration, and oversampling of demonstration data -- which forms the three phases of our model-based RL framework.
Deep Reinforcement Learning
Model-based Reinforcement Learning
+2
PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models
Generalizable 3D part segmentation is important but challenging in vision and robotics.
DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation
We empirically evaluate our method using an Allegro Hand to grasp novel objects in both simulation and real world.
On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning
Reinforcement Learning (RL) algorithms can solve challenging control problems directly from image observations, but they often require millions of environment interactions to do so.
Abstract-to-Executable Trajectory Translation for One-Shot Task Generalization
In the abstract environment, complex dynamics such as physical manipulation are removed, making abstract trajectories easier to generate.
Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds
We study how choices of input point cloud coordinate frames impact learning of manipulation skills from 3D point clouds.
LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds
Our method co-designs an efficient labeling process with semi/weakly supervised learning and is applicable to nearly any 3D semantic segmentation backbones.
Multi-skill Mobile Manipulation for Object Rearrangement
To tackle the entire task, prior work chains multiple stationary manipulation skills with a point-goal navigation skill, which are learned individually on subtasks.
Improving Policy Optimization with Generalist-Specialist Learning
Generalization in deep reinforcement learning over unseen environment variations usually requires policy learning over a large set of diverse training variations.
Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Search
Approximate convex decomposition aims to decompose a 3D shape into a set of almost convex components, whose convex hulls can then be used to represent the input shape.
Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics
Extensive experimental results suggest that: 1) on multi-stage tasks that are infeasible for the vanilla differentiable physics solver, our approach discovers contact points that efficiently guide the solver to completion; 2) on tasks where the vanilla solver performs sub-optimally or near-optimally, our contact point discovery method performs better than or on par with the manipulation performance obtained with handcrafted contact points.
From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation
We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand.
NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction
We demonstrate that NeRFusion achieves state-of-the-art quality on both large-scale indoor and small-scale object scenes, with substantially faster reconstruction than NeRF and other recent methods.
TensoRF: Tensorial Radiance Fields
We demonstrate that applying traditional CP decomposition -- that factorizes tensors into rank-one components with compact vectors -- in our framework leads to improvements over vanilla NeRF.
Ranked #3 on
Novel View Synthesis
on X3D
Temporal Difference Learning for Model Predictive Control
Data-driven model predictive control has two key advantages over model-free methods: a potential for improved sample efficiency through model learning, and better performance as computational budget for planning increases.
ActiveZero: Mixed Domain Learning for Active Stereovision with Zero Annotation
First, we demonstrate the transferability of our method to out-of-distribution real data by using a mixed domain learning strategy.
FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimization
We approach text-to-image generation by combining the power of the retrained CLIP representation with an off-the-shelf image generator (GANs), optimizing in the latent space of GAN to find images that achieve maximum CLIP score with the given input text.
Ranked #46 on
Text-to-Image Generation
on MS COCO
MARVEL: Raster Manga Vectorization via Primitive-wise Deep Reinforcement Learning
Manga is a fashionable Japanese-style comic form that is composed of black-and-white strokes and is generally displayed as raster images on digital devices.
ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Here we propose SAPIEN Manipulation Skill Benchmark (ManiSkill) to benchmark manipulation skills over diverse objects in a full-physics simulator.
Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
Our method greatly improves stability and sample efficiency of ConvNets under augmentation, and achieves generalization results competitive with state-of-the-art methods for image-based RL in environments with unseen visuals.
O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning
Contrary to the vast literature in modeling, perceiving, and understanding agent-object (e. g., human-object, hand-object, robot-object) interaction in computer vision and robotics, very few past works have studied the task of object-object interaction, which also plays an important role in robotic manipulation and planning tasks.
Single RGB-D Camera Teleoperation for General Robotic Manipulation
We propose a teleoperation system that uses a single RGB-D camera as the human motion capture device.
Particle Cloud Generation with Message Passing Generative Adversarial Networks
We propose JetNet as a novel point-cloud-style dataset for the ML community to experiment with, and set MPGAN as a benchmark to improve upon for future generative models.
ArtCoder: An End-to-End Method for Generating Scanning-Robust Stylized QR Codes
Quick Response (QR) code is one of the most worldwide used two-dimensional codes.
One-shot Learning with Absolute Generalization
One-shot learning is proposed to make a pretrained classifier workable on a new dataset based on one labeled samples from each pattern.
CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds
For the first time, we propose a unified framework that can handle 9DoF pose tracking for novel rigid object instances as well as per-part pose tracking for articulated objects from known categories.
PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
Experimental results suggest that 1) RL-based approaches struggle to solve most of the tasks efficiently; 2) gradient-based approaches, by optimizing open-loop control sequences with the built-in differentiable physics engine, can rapidly find a solution within tens of iterations, but still fall short on multi-stage tasks that require long-term planning.
MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis.
GNeRF: GAN-based Neural Radiance Field without Posed Camera
We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field (NeRF) reconstruction for the complex scenarios with unknown and even randomly initialized camera poses.
NeuTex: Neural Texture Mapping for Volumetric Neural Rendering
We achieve this by introducing a 3D-to-2D texture mapping (or surface parameterization) network into volumetric representations.
DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinates
Given a collection of 3D meshes of a category and their deformation handles (control points), our method learns a set of meta-handles for each shape, which are represented as combinations of the given handles.
Medical Robots for Infectious Diseases: Lessons and Challenges from the COVID-19 Pandemic
Medical robots can play an important role in mitigating the spread of infectious diseases and delivering quality care to patients during the COVID-19 pandemic.
Robotics
Semantically Robust Unpaired Image Translation for Data with Unmatched Semantics Statistics
Many applications of unpaired image-to-image translation require the input contents to be preserved semantically during translations.
Compositionally Generalizable 3D Structure Prediction
However, there remains a much more difficult and under-explored issue on how to generalize the learned skills over unseen object categories that have very different shape geometry distributions.
Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous GNNs
Current graph neural networks (GNNs) lack generalizability with respect to scales (graph sizes, graph diameters, edge weights, etc..) when solving many graph analysis problems.
An End-to-end Method for Producing Scanning-robust Stylized QR Codes
Quick Response (QR) code is one of the most worldwide used two-dimensional codes.~Traditional QR codes appear as random collections of black-and-white modules that lack visual semantics and aesthetic elements, which inspires the recent works to beautify the appearances of QR codes.
Rearrangement: A Challenge for Embodied AI
In the rearrangement task, the goal is to bring a given physical environment into a specified state.
Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals
We study how to learn a policy with compositional generalizability.
Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous Graph Neural Networks
Current graph neural networks (GNNs) lack generalizability with respect to scales (graph sizes, graph diameters, edge weights, etc..) when solving many graph analysis problems.
BiPointNet: Binary Neural Network for Point Clouds
To alleviate the resource constraint for real-time point cloud applications that run on edge devices, in this paper we present BiPointNet, the first model binarization approach for efficient deep learning on point clouds.
Photon-Driven Neural Path Guiding
To fully make use of our deep neural network, we partition the scene space into an adaptive hierarchical grid, in which we apply our network to reconstruct high-quality sampling distributions for any local region in the scene.
Weakly-supervised 3D Shape Completion in the Wild
3D shape completion for real data is important but challenging, since partial point clouds acquired by real-world sensors are usually sparse, noisy and unaligned.
Deep Keypoint-Based Camera Pose Estimation with Geometric Constraints
Estimating relative camera poses from consecutive frames is a fundamental problem in visual odometry (VO) and simultaneous localization and mapping (SLAM), where classic methods consisting of hand-crafted features and sampling-based outlier rejection have been a dominant choice for over a decade.
Balance Scene Learning Mechanism for Offshore and Inshore Ship Detection in SAR Images
Huge imbalance of different scenes' sample numbers seriously reduces Synthetic Aperture Radar (SAR) ship detection accuracy.
Meshing Point Clouds with Predicted Intrinsic-Extrinsic Ratio Guidance
We are interested in reconstructing the mesh representation of object surfaces from point clouds.
SofGAN: A Portrait Image Generator with Dynamic Styling
To address this issue, we propose a SofGAN image generator to decouple the latent space of portraits into two subspaces: a geometry space and a texture space.
Rethinking Sampling in 3D Point Cloud Generative Adversarial Networks
We further study how different evaluation metrics weigh the sampling pattern against the geometry and propose several perceptual metrics forming a sampling spectrum of metrics.
Deep Photon Mapping
This network is easy to incorporate in many previous photon mapping methods (by simply swapping the kernel density estimator) and can produce high-quality reconstructions of complex global illumination effects like caustics with an order of magnitude fewer photons compared to previous photon mapping methods.
MangaGAN: Unpaired Photo-to-Manga Translation Based on The Methodology of Manga Drawing
Manga is a world popular comic form originated in Japan, which typically employs black-and-white stroke lines and geometric exaggeration to describe humans' appearances, poses, and actions.
SAPIEN: A SimulAted Part-based Interactive ENvironment
To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable.
Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories
We address the problem of discovering 3D parts for objects in unseen categories.
Regret Bounds for Discounted MDPs
We give motivations and derive lower and upper bounds for such measures.
Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
In contrast, we propose adaptive thin volumes (ATVs); in an ATV, the depth hypothesis of each plane is spatially varying, which adapts to the uncertainties of previous per-pixel depth predictions.
Ranked #14 on
3D Reconstruction
on DTU
StructEdit: Learning Structural Shape Variations
Learning to encode differences in the geometry and (topological) structure of the shapes of ordinary objects is key to generating semantically plausible variations of a given shape, transferring edits from one shape to another, and many other applications in 3D content creation.
Normal Assisted Stereo Depth Estimation
We couple the learning of a multi-view normal estimation module and a multi-view depth estimation module.
State Alignment-based Imitation Learning
Consider an imitation learning problem that the imitator and the expert have different dynamics models.
Information-Theoretic Local Minima Characterization and Regularization
Recent advances in deep learning theory have evoked the study of generalizability across different local minima of deep neural networks (DNNs).
S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes
Grasping is among the most fundamental and long-lasting problems in robotics study.
Multi-view PointNet for 3D Scene Understanding
Fusion of 2D images and 3D point clouds is important because information from dense images can enhance sparse point clouds.
Multi-task Batch Reinforcement Learning with Metric Learning
To perform well, the policy must infer the task identity from collected transitions by modelling its dependency on states, actions and rewards.
Model Imitation for Model-Based Reinforcement Learning
Based on the claim, we propose to learn the transition model by matching the distributions of multi-step rollouts sampled from the transition model and the real ones via WGAN.
Model-based Reinforcement Learning
reinforcement-learning
+2
Pre-training as Batch Meta Reinforcement Learning with tiMe
Combining ideas from Batch RL and Meta RL, we propose tiMe, which learns distillation of multiple value functions and MDP embeddings from only existing data.
Adversarial shape perturbations on 3D point clouds
The importance of training robust neural network grows as 3D data is increasingly utilized in deep learning for vision tasks in robotics, drone control, and autonomous driving.
Mapping State Space using Landmarks for Universal Goal Reaching
An agent that has well understood the environment should be able to apply its skills for any given goals, leading to the fundamental problem of learning the Universal Value Function Approximator (UVFA).
Point-Based Multi-View Stereo Network
More specifically, our method predicts the depth in a coarse-to-fine manner.
StructureNet: Hierarchical Graph Networks for 3D Shape Generation
We introduce StructureNet, a hierarchical graph network which (i) can directly encode shapes represented as such n-ary graphs; (ii) can be robustly trained on large and complex shape families; and (iii) can be used to generate a great diversity of realistic structured shape geometries.
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
A human-specified design choice in domain randomization is the form and parameters of the distribution of simulated environments.
DeepSpline: Data-Driven Reconstruction of Parametric Curves and Surfaces
Reconstruction of geometry based on different input modes, such as images or point clouds, has been instrumental in the development of computer aided design and computer graphics.
Extending Adversarial Attacks and Defenses to Deep 3D Point Cloud Classifiers
We present a preliminary evaluation of adversarial attacks on deep 3D point cloud classifiers, namely PointNet and PointNet++, by evaluating both white-box and black-box adversarial attacks that were proposed for 2D images and extending those attacks to reduce the perceptibility of the perturbations in 3D space.
A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
To reduce memory footprint and run-time latency, techniques such as neural network pruning and binarization have been explored separately.
PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding
We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information.
Ranked #3 on
3D Instance Segmentation
on PartNet
Adversarial Defense by Stratified Convolutional Sparse Coding
We propose an adversarial defense method that achieves state-of-the-art performance among attack-agnostic adversarial defense methods while also maintaining robustness to input resolution, scale of adversarial perturbation, and scale of dataset size.
ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving
Specifically, we first segment each car with a pre-trained Mask R-CNN, and then regress towards its 3D pose and shape based on a deformable 3D car model with or without using semantic keypoints.
Towards Fast and Energy-Efficient Binarized Neural Network Inference on FPGA
Thus there does exist redundancy that can be exploited to further reduce the amount of on-chip computations.
Transfer Value or Policy? A Value-centric Framework Towards Transferrable Continuous Reinforcement Learning
In this paper, we investigate the problem of transfer learning across environments with different dynamics while accomplishing the same task in the continuous control domain.
Deep Part Induction from Articulated Object Pairs
In this paper, we explore how the observation of different articulation states provides evidence for part structure and motion of 3D objects.
Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
Binary neural networks (BNN) have been studied extensively since they run dramatically faster at lower memory and power consumption than floating-point networks, thanks to the efficiency of bit operations.
Geometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning
In addition, we also find that a progressive training strategy can foster a better neural network for the video recognition task than blindly pooling the distinct sources of geometry cues together.
Deep Functional Dictionaries: Learning Consistent Semantic Structures on 3D Models from Functions
Even though our shapes have independent discretizations and no functional correspondences are provided, the network is able to generate latent bases, in a consistent order, that reflect the shared semantic structure among the shapes.
View Extrapolation of Human Body from a Single Image
We study how to synthesize novel views of human body from a single image.
Robust Watertight Manifold Surface Generation Method for ShapeNet Models
In this paper, we describe a robust algorithm for 2-Manifold generation of various kinds of ShapeNet Models.
Computational Geometry
Frustum PointNets for 3D Object Detection from RGB-D Data
In this work, we study 3D object detection from RGB-D data in both indoor and outdoor scenes.
Ranked #1 on
Object Localization
on KITTI Pedestrians Easy
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database.
ComplementMe: Weakly-Supervised Component Suggestions for 3D Modeling
The combinatorial nature of part arrangements poses another challenge, since the retrieval network is not a function: several complements can be appropriate for the same input.
Graphics I.3.5
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
By exploiting metric space distances, our network is able to learn local features with increasing contextual scales.
Ranked #2 on
Semantic Segmentation
on Toronto-3D L002
Learning Non-Lambertian Object Intrinsics across ShapeNet Categories
Rendered with realistic environment maps, millions of synthetic images of objects and their corresponding albedo, shading, and specular ground-truth images are used to train an encoder-decoder CNN.
Beyond Holistic Object Recognition: Enriching Image Understanding with Part States
Important high-level vision tasks such as human-object interaction, image captioning and robotic manipulation require rich semantic descriptions of objects at part level.
A Point Set Generation Network for 3D Object Reconstruction from a Single Image
Our final solution is a conditional shape sampler, capable of predicting multiple plausible 3D point clouds from an input image.
Ranked #2 on
3D Reconstruction
on Data3D−R2N2
(using extra training data)
3D Object Reconstruction From A Single Image
3D Reconstruction
SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation
To enable the prediction of vertex functions on them by convolutional neural networks, we resort to spectral CNN method that enables weight sharing by parameterizing kernels in the spectral domain spanned by graph laplacian eigenbases.
Ranked #57 on
3D Part Segmentation
on ShapeNet-Part
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Point cloud is an important type of geometric data structure.
Ranked #1 on
3D Face Reconstruction
on !(()&&!|*|*|
Learning Shape Abstractions by Assembling Volumetric Primitives
We present a learning framework for abstracting complex shapes by learning to assemble objects using 3D volumetric primitives.
Multilinear Hyperplane Hashing
Despite its successful progress in classic point-to-point search, there are few studies regarding point-to-hyperplane search, which has strong practical capabilities of scaling up in many applications like active learning with SVMs.
FPNN: Field Probing Neural Networks for 3D Data
Each field probing filter is a set of probing points --- sensors that perceive the space.
Ranked #5 on
3D Object Recognition
on ModelNet40