Search Results for author:
Found 82 papers, 30 papers with code
NeRF-MAE : Masked AutoEncoders for Self Supervised 3D representation Learning for Neural Radiance Fields
Given the capabilities of neural fields in densely representing a 3D scene from 2D images, we ask the question: Can we scale their self-supervised pretraining, specifically using masked autoencoders, to generate effective 3D representations from posed RGB images.
Zero-Shot Multi-Object Shape Completion
We present a 3D shape completion method that recovers the complete geometry of multiple objects in complex scenes from a single RGB-D image.
Understanding Video Transformers via Universal Concept Discovery
Concretely, we seek to explain the decision-making process of video transformers based on high-level, spatiotemporal concepts that are automatically discovered.
Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?
To answer this question, we consider a set of tailored offline reinforcement learning datasets that exhibit causal ambiguity and assess the ability of active sampling techniques to reduce causal confusion at evaluation.
NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes
NeO 360's representation allows us to learn from a large collection of unbounded 3D scenes while offering generalizability to new views and novel scenes from as few as a single image during inference.
Ranked #1 on
Generalizable Novel View Synthesis
on NERDS 360
Robust Self-Supervised Extrinsic Self-Calibration
Autonomous vehicles and robots need to operate over a wide variety of scenarios in order to complete tasks efficiently and safely.
Towards Zero-Shot Scale-Aware Monocular Depth Estimation
Monocular depth estimation is scale-ambiguous, and thus requires scale supervision to produce metric predictions.
Multi-Object Manipulation via Object-Centric Neural Scattering Functions
Learned visual dynamics models have proven effective for robotic manipulation tasks.
HomE: Homography-Equivariant Video Representation Learning
In this work, we propose a novel method for representation learning of multi-view videos, where we explicitly model the representation space to maintain Homography Equivariance (HomE).
NeRFuser: Large-Scale Scene Representation by NeRF Fusion
A practical benefit of implicit visual representations like Neural Radiance Fields (NeRFs) is their memory efficiency: large scenes can be efficiently stored and shared as small neural nets instead of collections of images.
DeLiRa: Self-Supervised Depth, Light, and Radiance Fields
In this work, we propose to use the multi-view photometric objective from the self-supervised depth estimation literature as a geometric regularizer for volumetric rendering, significantly improving novel view synthesis without requiring additional information.
Object Discovery from Motion-Guided Tokens
Object discovery -- separating objects from the background without manual labels -- is a fundamental open challenge in computer vision.
Viewpoint Equivariance for Multi-View 3D Object Detection
We design view-conditioned queries at the output level, which enables the generation of multiple virtual frames during training to learn viewpoint equivariance by enforcing multi-view consistency.
In-Distribution Barrier Functions: Self-Supervised Policy Filters that Avoid Out-of-Distribution States
Learning-based control approaches have shown great promise in performing complex tasks directly from high-dimensional perception data for real robotic systems.
Breaking the "Object" in Video Object Segmentation
Yet, this important phenomenon is largely absent from existing video object segmentation (VOS) benchmarks.
ROAD: Learning an Implicit Recursive Octree Auto-Decoder to Efficiently Encode 3D Shapes
Compact and accurate representations of 3D shapes are central to many perception and robotics tasks.
Photo-realistic Neural Domain Randomization
In this paper, we show that the recent progress in neural rendering enables a new unified approach we call Photo-realistic Neural Domain Randomization (PNDR).
Depth Is All You Need for Monocular 3D Detection
Our methods leverage commonly available LiDAR or RGB videos during training time to fine-tune the depth representation, which leads to improved 3D detectors.
RAP: Risk-Aware Prediction for Robust Planning
Robust planning in interactive scenarios requires predicting the uncertain future to make risk-aware decisions.
Depth Field Networks for Generalizable Multi-view Scene Representation
Modern 3D computer vision leverages learning to boost geometric reasoning, mapping image data to classical structures such as cost volumes or epipolar constraints to improve matching.
ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Optimization
A novel disentangled shape and appearance database of priors is first learned to embed objects in their respective shape and appearance space.
3D Shape Reconstruction From A Single 2D Image
6D Pose Estimation
+4
Neural Groundplans: Persistent Neural Scene Representations from a Single Image
We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene.
Revisiting the "Video" in Video-Language Understanding
Building on recent progress in self-supervised image-language models, we revisit this question in the context of video and language tasks.
Ranked #1 on
Video Question Answering
on MSR-VTT-MC
Control-Aware Prediction Objectives for Autonomous Driving
Autonomous vehicle software is typically structured as a modular pipeline of individual components (e. g., perception, prediction, and planning) to help separate concerns into interpretable sub-tasks.
Multi-Frame Self-Supervised Depth with Transformers
Experiments on the KITTI and DDAD datasets show that our DepthFormer architecture establishes a new state of the art in self-supervised monocular depth estimation, and is even competitive with highly specialized supervised single-frame architectures.
Object Permanence Emerges in a Random Walk along Memory
This paper proposes a self-supervised objective for learning representations that localize objects under occlusion - a property known as object permanence.
Learning Optical Flow, Depth, and Scene Flow without Real-World Labels
However, the simultaneous self-supervised learning of depth and scene flow is ill-posed, as there are infinitely many combinations that result in the same 3D point.
Discovering Objects that Can Move
Our experiments demonstrate that, despite only capturing a small subset of the objects that move, this signal is enough to generalize to segment both moving and static instances of dynamic objects.
Dynamics-Aware Comparison of Learned Reward Functions
The ability to learn reward functions plays an important role in enabling the deployment of intelligent agents in the real world.
Self-Supervised Camera Self-Calibration from Video
Camera calibration is integral to robotics and computer vision algorithms that seek to infer geometric properties of the scene from visual input streams.
Self-supervised Learning is More Robust to Dataset Imbalance
Third, inspired by the theoretical insights, we devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets with several evaluation criteria, closing the small gap between balanced and imbalanced datasets with the same number of examples.
Ranked #9 on
Long-tail Learning
on CIFAR-10-LT (ρ=100)
Warp-Refine Propagation: Semi-Supervised Auto-labeling via Cycle-consistency
Deep learning models for semantic segmentation rely on expensive, large-scale, manually annotated datasets.
Ranked #34 on
Semantic Segmentation
on NYU Depth v2
Is Pseudo-Lidar needed for Monocular 3D Object detection?
Recent progress in 3D object detection from single images leverages monocular depth estimation as a way to produce 3D pointclouds, turning cameras into pseudo-lidar sensors.
Ranked #1 on
Monocular 3D Object Detection
on KITTI Pedestrian Moderate
(using extra training data)
Hierarchical Lovasz Embeddings for Proposal-Free Panoptic Segmentation
We use a hierarchical Lovasz hinge loss to learn a low-dimensional embedding space structured into a unified semantic and instance hierarchy without requiring separate network branches or object proposals.
Hierarchical Lovász Embeddings for Proposal-free Panoptic Segmentation
We use a hierarchical Lov\'asz hinge loss to learn a low-dimensional embedding space structured into a unified semantic and instance hierarchy without requiring separate network branches or object proposals.
Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
Despite the empirical successes, theoretical foundations are limited -- prior analyses assume conditional independence of the positive pairs given the same class label, but recent empirical applications use heavily correlated positive pairs (i. e., data augmentations of the same image).
Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty
Reasoning about the future behavior of other agents is critical to safe robot navigation.
Full Surround Monodepth from Multiple Cameras
In this work, we extend monocular self-supervised depth and ego-motion estimation to large-baseline multi-camera rigs.
Geometric Unsupervised Domain Adaptation for Semantic Segmentation
Simulators can efficiently generate large amounts of labeled synthetic data with perfect supervision for hard-to-label tasks like semantic segmentation.
Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion
Estimating scene geometry from data obtained with cost-effective sensors is key for robots and self-driving cars.
Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark
We present the design of a centralized benchmark for Reinforcement Learning which can help measure Sample Efficiency and Generalization in Reinforcement Learning by doing end to end evaluation of the training and rollout phases of thousands of user submitted code bases in a scalable way.
Learning to Track with Object Permanence
In this work, we introduce an end-to-end trainable approach for joint object detection and tracking that is capable of such reasoning.
Monocular Depth Estimation for Soft Visuotactile Sensors
Fluid-filled soft visuotactile sensors such as the Soft-bubbles alleviate key challenges for robust manipulation, as they enable reliable grasps along with the ability to obtain high-resolution sensory feedback on contact geometry and forces.
Bayesian Meta-Learning for Few-Shot 3D Shape Completion
Estimating the 3D shape of real-world objects is a key perceptual challenge.
Discovering Avoidable Planner Failures of Autonomous Vehicles using Counterfactual Analysis in Behaviorally Diverse Simulation
Automated Vehicles require exhaustive testing in simulation to detect as many safety-critical failures as possible before deployment on public roads.
Behaviorally Diverse Traffic Simulation via Reinforcement Learning
Traffic simulators are important tools in autonomous driving development.
Monocular Differentiable Rendering for Self-Supervised 3D Object Detection
3D object detection from monocular images is an ill-posed problem due to the projective entanglement of depth and scale.
3D Object Detection
3D Object Detection From Monocular Images
+5
MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control
Reasoning about human motion is a core component of modern human-robot interactive systems.
Risk-Sensitive Sequential Action Control with Multi-Modal Human Trajectory Forecasting for Safe Crowd-Robot Interaction
This paper presents a novel online framework for safe crowd-robot interaction based on risk-sensitive stochastic optimal control, wherein the risk is modeled by the entropic risk measure.
Driving Through Ghosts: Behavioral Cloning with False Positives
In this work, we propose a behavioral cloning approach that can safely leverage imperfect perception without being conservative.
Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion
Self-supervised learning has emerged as a powerful tool for depth and ego-motion estimation, leading to state-of-the-art results on benchmark datasets.
PillarFlow: End-to-end Birds-eye-view Flow Estimation for Autonomous Driving
In autonomous driving, accurately estimating the state of surrounding obstacles is critical for safe and robust path planning.
Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
To address driving in near-accident scenarios, we propose a hierarchical reinforcement and imitation learning (H-ReIL) approach that consists of low-level policies learned by IL for discrete driving modes, and a high-level policy learned by RL that switches between different driving modes.
Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization
Real-world large-scale datasets are heteroskedastic and imbalanced -- labels have varying levels of uncertainty and label distributions are long-tailed.
Ranked #11 on
Image Classification
on WebVision-1000
Differentiable Rendering: A Survey
Deep neural networks (DNNs) have shown remarkable performance improvements on vision-related tasks such as object detection or image segmentation.
It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction
In this work, we present Predicted Endpoint Conditioned Network (PECNet) for flexible human trajectory prediction.
Ranked #1 on
Multi Future Trajectory Prediction
on ETH/UCY
Spatio-Temporal Graph for Video Captioning with Knowledge Distillation
In this paper, we propose a novel spatio-temporal graph model for video captioning that exploits object interactions in space and time.
Semantically-Guided Representation Learning for Self-Supervised Monocular Depth
Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions.
Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction
In addition, we introduce a new dataset designed specifically for autonomous-driving scenarios in areas with dense pedestrian populations: the Stanford-TRI Intent Prediction (STIP) dataset.
Self-Supervised 3D Keypoint Learning for Ego-motion Estimation
Detecting and matching robust viewpoint-invariant keypoints is critical for visual SLAM and Structure-from-Motion.
Real-Time Panoptic Segmentation from Dense Detections
Panoptic segmentation is a complex full scene parsing task requiring simultaneous instance and semantic segmentation at high resolution.
Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors
We present an automatic annotation pipeline to recover 9D cuboids and 3D shapes from pre-trained off-the-shelf 2D detectors and sparse LIDAR data.
Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision
In contrast to the previous work that aims to solve either the task of pose prediction or trajectory forecasting in isolation, we propose a framework to unify the two problems and address the practically useful task of pedestrian locomotion prediction in the wild.
Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical Models
With this model we generate a diverse, realistic, and physically plausible dataset of human action videos, called PHAV for "Procedural Human Action Videos".
Two Stream Networks for Self-Supervised Ego-Motion Estimation
Learning depth and camera ego-motion from raw unlabeled RGB video streams is seeing exciting progress through self-supervision from strong geometric cues.
Robust Semi-Supervised Monocular Depth Estimation with Reprojected Distances
Dense depth estimation from a single image is a key problem in computer vision, with exciting applications in a multitude of robotic tasks.
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
Deep learning algorithms can fare poorly when the training dataset suffers from heavy class-imbalance but the testing criterion requires good generalization on less frequent classes.
Ranked #4 on
Long-tail learning with class descriptors
on CUB-LT
An Attention-based Recurrent Convolutional Network for Vehicle Taillight Recognition
Vehicle taillight recognition is an important application for automated driving, especially for intent prediction of ado vehicles and trajectory planning of the ego vehicle.
3D Packing for Self-Supervised Monocular Depth Estimation
Although cameras are ubiquitous, robotic platforms typically rely on active sensors like LiDAR for direct 3D perception.
Exploring the Limitations of Behavior Cloning for Autonomous Driving
Driving requires reacting to a wide variety of complex environment conditions and agent behaviors.
Ranked #17 on
Autonomous Driving
on CARLA Leaderboard
ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape
We present a deep learning method for end-to-end monocular 3D object detection and metric shape retrieval.
Learning to Fuse Things and Stuff
We propose an end-to-end learning approach for panoptic segmentation, a novel task unifying instance (things) and semantic (stuff) segmentation.
Ranked #26 on
Panoptic Segmentation
on Cityscapes val
(using extra training data)
SPIGAN: Privileged Adversarial Learning from Simulation
Deep Learning for Computer Vision depends mainly on the source of supervision. Photo-realistic simulators can generate large-scale automatically labeled syntheticdata, but introduce a domain gap negatively impacting performance.
SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation
Both contributions provide significant performance gains over the state-of-the-art in self-supervised depth and pose estimation on the public KITTI benchmark.
Procedural Generation of Videos to Train Deep Action Recognition Networks
Deep learning for human action recognition in videos is making significant progress, but is slowed down by its dependency on expensive manual labeling of large video collections.
Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition
Action recognition in videos is a challenging task due to the complexity of the spatio-temporal patterns to model and the difficulty to acquire and learn on large quantities of video data.
Virtual Worlds as Proxy for Multi-Object Tracking Analysis
We provide quantitative experimental evidence suggesting that (i) modern deep learning algorithms pre-trained on real data behave similarly in real and virtual worlds, and (ii) pre-training on virtual data improves performance.
Online Domain Adaptation for Multi-Object Tracking
We quantitatively measure the benefit of our domain adaptation strategy on the KITTI tracking benchmark and on a new dataset (PASCAL-to-KITTI) we introduce to study the domain mismatch problem in MOT.
Deep Fishing: Gradient Features from Deep Nets
Convolutional Networks (ConvNets) have recently improved image recognition performance thanks to end-to-end learning of deep feed-forward models from raw pixels.
Online Learning to Sample
Stochastic Gradient Descent (SGD) is one of the most widely used techniques for online optimization in machine learning.
Self-Learning Camera: Autonomous Adaptation of Object Detectors to Unlabeled Video Streams
In this paper, we address the problem of self-learning detectors in an autonomous manner, i. e. (i) detectors continuously updating themselves to efficiently adapt to streaming data sources (contrary to transductive algorithms), (ii) without any labeled data strongly related to the target data stream (contrary to self-paced learning), and (iii) without manual intervention to set and update hyper-parameters.
