Search Results for author:
Found 123 papers, 90 papers with code
Online Continual Learning Without the Storage Constraint
Online continual learning (OCL) research has primarily focused on mitigating catastrophic forgetting with fixed and limited storage allocation throughout the agent's lifetime.
Monocular Visual-Inertial Depth Estimation
We evaluate on the TartanAir and VOID datasets, observing up to 30% reduction in inverse RMSE with dense scale alignment relative to performing just global alignment alone.
Guaranteed Conservation of Momentum for Learning Particle-based Fluid Dynamics
We present a novel method for guaranteeing linear momentum in learned physics simulations.
Improving information retention in large scale online continual learning
The typical approach to address information retention (the ability to retain previous knowledge) is keeping a replay buffer of a fixed size and computing gradients using a mixture of new data and the replay buffer.
Differentiable Simulation of Soft Multi-body Systems
We present a method for differentiable simulation of soft articulated bodies.
Unsupervised Contrastive Domain Adaptation for Semantic Segmentation
Semantic segmentation models struggle to generalize in the presence of domain shift.
Dancing under the stars: video denoising in starlight
To enable this, we develop a GAN-tuned physics-based noise model to more accurately represent camera noise at the lowest light levels.
Global Tracking Transformers
The transformer encodes object features from all frames, and uses trajectory queries to group them into trajectories.
Ranked #8 on
Multi-Object Tracking
on SportsMOT
Language-driven Semantic Segmentation
We present LSeg, a novel model for language-driven semantic image segmentation.
Ranked #1 on
Few-Shot Semantic Segmentation
on FSS-1000
MSeg: A Composite Dataset for Multi-domain Semantic Segmentation
We adopt zero-shot cross-dataset transfer as a benchmark to systematically evaluate a model's robustness and show that MSeg training yields substantially more robust models in comparison to training on individual datasets or naive mixing of datasets without the presented contributions.
Ranked #6 on
Semantic Segmentation
on ScanNetV2
Shape from Polarization for Complex Scenes in the Wild
We present a new data-driven approach with physics-based priors to scene-level normal estimation from a single polarization image.
Geometry Processing with Neural Fields
Most existing geometry processing algorithms use meshes as the default shape representation.
Non-deep Networks
This begs the question -- is it possible to build high-performing "non-deep" neural networks?
Learning High-Speed Flight in the Wild
Indeed, the subtasks are executed sequentially, leading to increased processing latency and a compounding of errors through the pipeline.
ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception
Unlike existing GPU hash maps, the ASH framework provides a versatile tensor interface, hiding low-level details from the users.
Scale-invariant Learning by Physics Inversion
We find that state-of-the-art training techniques are not well-suited to many problems that involve physical processes.
Neural Deep Equilibrium Solvers
A deep equilibrium (DEQ) model abandons traditional depth by solving for the fixed point of a single nonlinear layer $f_\theta$.
Efficient Differentiable Simulation of Articulated Bodies
We derive the gradients of the forward dynamics using spatial algebra and the adjoint method.
Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data
We argue that "online" continual learning, where data is a single continuous stream without task boundaries, enables evaluating both information retention and online learning efficacy.
Megaverse: Simulating Embodied Agents at One Million Experiences per Second
We present Megaverse, a new 3D simulation platform for reinforcement learning and embodied AI research.
Stabilizing Equilibrium Models by Jacobian Regularization
Deep equilibrium networks (DEQs) are a new class of models that eschews traditional depth in favor of finding the fixed point of a single nonlinear layer.
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
We introduce Habitat 2. 0 (H2. 0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios.
Training Graph Neural Networks with 1000 Layers
Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges.
Ranked #1 on
Node Property Prediction
on ogbn-proteins
A Measure of Research Taste
The presented measure, CAP, balances the impact of publications and their quantity, thus incentivizing researchers to consider whether a publication is a useful addition to the literature.
Enhancing Photorealism Enhancement
We confirm the benefits of our contributions in controlled experiments and report substantial gains in stability and realism in comparison to recent image-to-image translation methods and a variety of other baselines.
Learning to drive from a world on rails
This assumption greatly simplifies the learning problem, factorizing the dynamics into a nonreactive world model and a low-dimensional and compact forward model of the ego-vehicle.
Ranked #8 on
Autonomous Driving
on CARLA Leaderboard
Vision Transformers for Dense Prediction
We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks.
Ranked #11 on
Semantic Segmentation
on PASCAL Context
Large Batch Simulation for Deep Reinforcement Learning
We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work, realizing end-to-end training speeds of over 19, 000 frames of experience per second on a single GPU and up to 72, 000 frames per second on a single eight-GPU machine.
Probabilistic two-stage detection
We develop a probabilistic interpretation of two-stage object detection.
Ranked #38 on
Object Detection
on COCO test-dev
Self-supervised Geometric Perception
We present self-supervised geometric perception (SGP), the first general framework to learn a feature descriptor for correspondence matching without any ground-truth geometric model labels (e. g., camera poses, rigid transformations).
Simple multi-dataset detection
Experiments show our learned taxonomy outperforms a expert-designed taxonomy in all datasets.
Learning a unified label space
These labels span many diverse datasets with potentially inconsistent semantic labels.
Adaptive Surface Reconstruction With Multiscale Convolutional Kernels
We propose generalized convolutional kernels for 3D reconstruction with ConvNets from point clouds.
Point Transformer
For example, on the challenging S3DIS dataset for large-scale semantic scene segmentation, the Point Transformer attains an mIoU of 70. 4% on Area 5, outperforming the strongest prior model by 3. 3 absolute percentage points and crossing the 70% mIoU threshold for the first time.
Ranked #3 on
3D Semantic Segmentation
on STPLS3D
Stable View Synthesis
The core of SVS is view-dependent on-surface feature aggregation, in which directional feature vectors at each 3D point are processed to produce a new feature vector for a ray that maps this point into the new target view.
Rearrangement: A Challenge for Embodied AI
In the rearrangement task, the goal is to bring a given physical environment into a specified state.
Learning Quadrupedal Locomotion over Challenging Terrain
The trained controller has taken two generations of quadrupedal ANYmal robots to a variety of natural environments that are beyond the reach of prior published work in legged locomotion.
NeRF++: Analyzing and Improving Neural Radiance Fields
Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings, including 360 capture of bounded scenes and forward-facing capture of bounded and unbounded scenes.
OpenBot: Turning Smartphones into Robots
We develop a software stack that allows smartphones to use this body for mobile operation and demonstrate that the system is sufficiently powerful to support advanced robotics workloads such as person following and real-time autonomous navigation in unstructured environments.
Free View Synthesis
We present a method for novel view synthesis from input images that are freely distributed around a scene.
Drinking from a Firehose: Continual Learning with Web-scale Natural Language
Collectively, the POLL problem setting, the Firehose datasets, and the ConGraD algorithm enable a complete benchmark for reproducible research on web-scale continual learning.
Dynamic Low-light Imaging with Quanta Image Sensors
QIS are single-photon image sensors with photon counting capabilities.
Scaling Imitation Learning in Minecraft
Imitation learning is a powerful family of techniques for learning sensorimotor coordination in immersive environments.
Scalable Differentiable Physics for Learning and Control
Differentiable physics is a powerful approach to learning and control problems that involve physical objects and environments.
Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning
In this work we aim to solve this problem by optimizing the efficiency and resource utilization of reinforcement learning algorithms instead of relying on distributed computation.
Multiscale Deep Equilibrium Models
These simultaneously-learned multi-resolution features allow us to train a single model on a diverse set of tasks and loss functions, such as using a single MDEQ to perform both image classification and semantic segmentation.
Ranked #39 on
Semantic Segmentation
on Cityscapes val
Deep Drone Acrobatics
In this paper, we propose to learn a sensorimotor policy that enables an autonomous quadrotor to fly extreme acrobatic maneuvers with only onboard sensing and computation.
Robotics
Auto-decoding Graphs
Graph-based normalizing flows are used to sample latent codes from the distribution learned by the auto-decoder.
High-dimensional Convolutional Networks for Geometric Pattern Recognition
Many problems in science and engineering can be formulated in terms of geometric patterns in high-dimensional spaces.
Lagrangian Fluid Simulation with Continuous Convolutions
We present an approach to Lagrangian fluid simulation with a new type of convolutional network.
Exploring Self-attention for Image Recognition
Recent work has shown that self-attention can serve as a basic building block for image recognition models.
Learning to Guide Random Search
In other words, we jointly learn the manifold and optimize the function.
Deep Global Registration
We present Deep Global Registration, a differentiable framework for pairwise registration of real-world 3D scans.
Ranked #2 on
Point Cloud Registration
on KITTI (FCGF setting)
Tracking Objects as Points
Nowadays, tracking is dominated by pipelines that perform object detection followed by temporal association, also known as tracking-by-detection.
Ranked #3 on
Multiple Object Tracking
on KITTI Tracking test
Learning to Control PDEs with Differentiable Physics
Predicting outcomes and planning interactions with the physical world are long-standing goals for machine learning.
Learning by Cheating
We first train an agent that has access to privileged information.
Ranked #12 on
Autonomous Driving
on CARLA Leaderboard
Differentiable Cloth Simulation for Inverse Problems
We propose a differentiable cloth simulator that can be embedded as a layer in deep neural networks.
Fully Convolutional Geometric Features
Extracting geometric features from 3D scans or point clouds is the first step in applications such as registration, reconstruction, and tracking.
Ranked #1 on
3D Feature Matching
on 3DMatch Benchmark
Deep Equilibrium Models
We present a new approach to modeling sequential data: the deep equilibrium model (DEQ).
Ranked #28 on
Language Modelling
on Penn Treebank (Word Level)
Consensus Maximization Tree Search Revisited
First, we show that the consensus maximization tree structure used previously actually contains paths that connect nodes at both adjacent and non-adjacent levels.
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
In particular, we propose a robust training objective that is invariant to changes in depth range and scale, advocate the use of principled multi-objective learning to combine data from different sources, and highlight the importance of pretraining encoders on auxiliary tasks.
Ranked #2 on
Depth Estimation
on eBDtheque
The Limited Multi-Label Projection Layer
We propose the Limited Multi-Label (LML) projection layer as a new primitive operation for end-to-end learning systems.
High Speed and High Dynamic Range Video with an Event Camera
In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors.
Does computer vision matter for action?
Thus the central question of our work: Does computer vision matter for action?
Zoom To Learn, Learn To Zoom
We show how to obtain the ground-truth data with optically zoomed images and contribute a dataset, SR-RAW, for real-world computational zoom.
What Do Single-view 3D Reconstruction Networks Learn?
Convolutional networks for single-view object reconstruction have shown impressive performance and have become a popular subject of research.
Ranked #1 on
3D Reconstruction
on 300W
Deep Layers as Stochastic Solvers
In particular, we show that a forward pass through a standard dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex optimization objective with a single iteration of a $\tau$-nice Proximal Stochastic Gradient method.
Events-to-Video: Bringing Modern Computer Vision to Event Cameras
Since the output of event cameras is fundamentally different from conventional cameras, it is commonly accepted that they require the development of specialized algorithms to accommodate the particular nature of events.
Habitat: A Platform for Embodied AI Research
We present Habitat, a platform for research in embodied artificial intelligence (AI).
Ranked #2 on
PointGoal Navigation
on Gibson PointGoal Navigation
Benchmarking Classic and Learned Navigation in Complex 3D Environments
However, this new line of work is largely disconnected from well-established classic navigation approaches.
Learning agile and dynamic motor skills for legged robots
In the present work, we introduce a method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system, thereby leveraging fast, automated, and cost-effective data generation schemes.
Motion Perception in Reinforcement Learning with Dynamic Objects
In dynamic environments, learned controllers are supposed to take motion into account when selecting the action to be taken.
Assessing Generalization in Deep Reinforcement Learning
Our aim is to catalyze community-wide progress on generalization in deep RL.
Out-of-Distribution Generalization
reinforcement-learning
+1
Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search
We present a learning-based approach to computing solutions for certain NP-hard problems.
Trellis Networks for Sequence Modeling
On the other hand, we show that truncated recurrent networks are equivalent to trellis networks with special sparsity structure in their weight matrices.
Multi-Task Learning as Multi-Objective Optimization
These algorithms are not directly applicable to large-scale learning problems since they scale poorly with the dimensionality of the gradients and the number of tasks.
Ranked #1 on
Multi-Task Learning
on CelebA
On Offline Evaluation of Vision-based Driving Models
We show that the correlation of offline evaluation with driving quality can be significantly improved by selecting an appropriate validation dataset and suitable offline metrics.
Deep Fundamental Matrix Estimation
We present an approach to robust estimation of fundamental matrices from noisy data contaminated by outliers.
On Evaluation of Embodied Navigation Agents
Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence.
Tangent Convolutions for Dense Prediction in 3D
Our approach is based on tangent convolutions - a new construction for convolutional networks on 3D data.
Ranked #7 on
3D Semantic Segmentation
on SensatUrban
Speech Denoising Convolutional Neural Network trained with Deep Feature Losses.
We present an end-to-end deep learning approach to denoising speech signals by processing the raw waveform directly.
Speech Denoising with Deep Feature Losses
We present an end-to-end deep learning approach to denoising speech signals by processing the raw waveform directly.
TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning
Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators.
Interactive Image Segmentation With Latent Diversity
The first is trained to synthesize a diverse set of plausible segmentations that conform to the user's input.
Ranked #10 on
Interactive Segmentation
on SBD
Learning to See in the Dark
Imaging in low light is challenging due to low photon count and low SNR.
Ranked #1 on
Image Denoising
on SID x300
Semi-parametric Image Synthesis
We present a semi-parametric approach to photographic image synthesis from semantic layouts.
Driving Policy Transfer via Modularity and Abstraction
Simulation can help end-to-end driving systems by providing a cheap, safe, and diverse training environment.
Deep Continuous Clustering
We present a clustering algorithm that performs nonlinear dimensionality reduction and clustering jointly.
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
Our results indicate that a simple convolutional architecture outperforms canonical recurrent networks such as LSTMs across a diverse range of tasks and datasets, while demonstrating longer effective memory.
Ranked #4 on
Music Modeling
on Nottingham
Semi-parametric Topological Memory for Navigation
We introduce a new memory architecture for navigation in previously unseen environments, inspired by landmark-based navigation in animals.
Open3D: A Modern Library for 3D Data Processing
The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python.
Convolutional Sequence Modeling Revisited
This paper revisits the problem of sequence modeling using convolutional architectures.
Ranked #66 on
Language Modelling
on WikiText-103
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
We present MINOS, a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments.
CARLA: An Open Urban Driving Simulator
We introduce CARLA, an open-source simulator for autonomous driving research.
End-to-end Driving via Conditional Imitation Learning
However, driving policies trained via imitation learning cannot be controlled at test time.
Colored Point Cloud Registration Revisited
We present an algorithm for tightly aligning two colored point clouds.
Learning to Inpaint for Image Compression
We study the design of deep architectures for lossy image compression.
Learning Compact Geometric Features
We present an approach to learning features that represent the local geometry around a point in an unstructured point cloud.
Ranked #9 on
Point Cloud Registration
on ETH (trained on 3DMatch)
Fast Image Processing with Fully-Convolutional Networks
Our approach uses a fully-convolutional network that is trained on input-output pairs that demonstrate the operator's action.
Photographic Image Synthesis with Cascaded Refinement Networks
We present an approach to synthesizing photographic images conditioned on semantic layouts.
Dilated Residual Networks
Convolutional networks for image classification progressively reduce resolution until the image is represented by tiny feature maps in which the spatial structure of the scene is no longer discernible.
Accurate Optical Flow via Direct Cost Volume Processing
We present an optical flow estimation approach that operates on the full four-dimensional cost volume.
Learning to Act by Predicting the Future
A model trained using the presented approach won the Full Deathmatch track of the Visual Doom AI Competition, which was held in previously unseen environments.
Fast Global Registration
Extensive experiments demonstrate that the presented approach matches or exceeds the accuracy of state-of-the-art global registration pipelines, while being at least an order of magnitude faster.
Playing for Data: Ground Truth from Computer Games
Recent progress in computer vision has been driven by high-capacity models trained on large datasets.
Dense Monocular Depth Estimation in Complex Dynamic Scenes
We present an approach to dense depth estimation from a single monocular camera that is moving through a dynamic scene.
Feature Space Optimization for Semantic Video Segmentation
We present an approach to long-range spatio-temporal regularization in semantic video segmentation.
Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids
The approach optimizes a classical optical flow objective over the full space of mappings between discrete grids.
A Large Dataset of Object Scans
We have created a dataset of more than ten thousand 3D scans of real objects.
Single-View Reconstruction via Joint Analysis of Image and Shape Collections
We present an approach to automatic 3D reconstruction of objects depicted in Web images.
Robust Nonrigid Registration by Convex Optimization
We present an approach to nonrigid registration of 3D surfaces.
Multi-Scale Context Aggregation by Dilated Convolutions
State-of-the-art models for semantic segmentation are based on adaptations of convolutional networks that had originally been designed for image classification.
Ranked #10 on
Semantic Segmentation
on CamVid
Robust Reconstruction of Indoor Scenes
We present an approach to indoor scene reconstruction from RGB-D video.
Depth Camera Tracking With Contour Cues
We present an approach for tracking camera pose in real time given a stream of depth images.
Learning to Propose Objects
We present an approach for highly accurate bottom-up object segmentation.
Fast MRF Optimization with Application to Depth Reconstruction
We describe a simple and fast algorithm for optimizing Markov random fields over images.
Simultaneous Localization and Calibration: Self-Calibration of Consumer Depth Cameras
We describe an approach for simultaneous localization and calibration of a stream of range images.
Variational Policy Search via Trajectory Optimization
In order to learn effective control policies for dynamical systems, policy search methods must be able to discover successful executions of the desired task.
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
In this paper, we consider fully connected CRF models defined on the complete set of pixels in an image.
Nonlinear Inverse Reinforcement Learning with Gaussian Processes
We present a probabilistic algorithm for nonlinear inverse reinforcement learning.
Feature Construction for Inverse Reinforcement Learning
The goal of inverse reinforcement learning is to find a reward function for a Markov decision process, given example traces from its optimal policy.
