Search Results for author: Vladlen Koltun

Found 114 papers, 80 papers with code

MSeg: A Composite Dataset for Multi-domain Semantic Segmentation

2 code implementations CVPR 2020 John Lambert, Zhuang Liu, Ozan Sener, James Hays, Vladlen Koltun

We adopt zero-shot cross-dataset transfer as a benchmark to systematically evaluate a model's robustness and show that MSeg training yields substantially more robust models in comparison to training on individual datasets or naive mixing of datasets without the presented contributions.

Instance Segmentation Panoptic Segmentation +1

Shape from Polarization for Complex Scenes in the Wild

1 code implementation21 Dec 2021 Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, Qifeng Chen

We present a new data-driven approach with physics-based priors to scene-level normal estimation from a single polarization image.

Geometry Processing with Neural Fields

1 code implementation NeurIPS 2021 Guandao Yang, Serge Belongie, Bharath Hariharan, Vladlen Koltun

Most existing geometry processing algorithms use meshes as the default shape representation.

Non-deep Networks

3 code implementations14 Oct 2021 Ankit Goyal, Alexey Bochkovskiy, Jia Deng, Vladlen Koltun

This begs the question -- is it possible to build high-performing "non-deep" neural networks?

Image Classification Real-Time Object Detection

Learning High-Speed Flight in the Wild

1 code implementation11 Oct 2021 Antonio Loquercio, Elia Kaufmann, René Ranftl, Matthias Müller, Vladlen Koltun, Davide Scaramuzza

Indeed, the subtasks are executed sequentially, leading to increased processing latency and a compounding of errors through the pipeline.

ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception

no code implementations1 Oct 2021 Wei Dong, Yixing Lao, Michael Kaess, Vladlen Koltun

Unlike existing GPU hash maps, the ASH framework provides a versatile tensor interface, hiding low-level details from the users.

Point Cloud Registration

Physical Gradients for Deep Learning

no code implementations30 Sep 2021 Philipp Holl, Vladlen Koltun, Nils Thuerey

Solving inverse problems, such as parameter estimation and optimal control, is a vital part of science.

Efficient Differentiable Simulation of Articulated Bodies

2 code implementations16 Sep 2021 Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin

We derive the gradients of the forward dynamics using spatial algebra and the adjoint method.

Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data

1 code implementation ICCV 2021 Zhipeng Cai, Ozan Sener, Vladlen Koltun

We argue that "online" continual learning, where data is a single continuous stream without task boundaries, enables evaluating both information retention and online learning efficacy.

Continual Learning

Megaverse: Simulating Embodied Agents at One Million Experiences per Second

1 code implementation17 Jul 2021 Aleksei Petrenko, Erik Wijmans, Brennan Shacklett, Vladlen Koltun

We present Megaverse, a new 3D simulation platform for reinforcement learning and embodied AI research.

Stabilizing Equilibrium Models by Jacobian Regularization

1 code implementation28 Jun 2021 Shaojie Bai, Vladlen Koltun, J. Zico Kolter

Deep equilibrium networks (DEQs) are a new class of models that eschews traditional depth in favor of finding the fixed point of a single nonlinear layer.

Language Modelling

Training Graph Neural Networks with 1000 Layers

4 code implementations14 Jun 2021 Guohao Li, Matthias Müller, Bernard Ghanem, Vladlen Koltun

Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges.

Graph Sampling Node Property Prediction

A Measure of Research Taste

no code implementations17 May 2021 Vladlen Koltun, David Hafner

The presented measure, CAP, balances the impact of publications and their quantity, thus incentivizing researchers to consider whether a publication is a useful addition to the literature.

Enhancing Photorealism Enhancement

no code implementations10 May 2021 Stephan R. Richter, Hassan Abu Alhaija, Vladlen Koltun

We confirm the benefits of our contributions in controlled experiments and report substantial gains in stability and realism in comparison to recent image-to-image translation methods and a variety of other baselines.

Image-to-Image Translation Translation

Learning to drive from a world on rails

1 code implementation ICCV 2021 Dian Chen, Vladlen Koltun, Philipp Krähenbühl

This assumption greatly simplifies the learning problem, factorizing the dynamics into a nonreactive world model and a low-dimensional and compact forward model of the ego-vehicle.

Autonomous Driving Model-based Reinforcement Learning

Vision Transformers for Dense Prediction

8 code implementations ICCV 2021 René Ranftl, Alexey Bochkovskiy, Vladlen Koltun

We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks.

Monocular Depth Estimation Semantic Segmentation

Large Batch Simulation for Deep Reinforcement Learning

1 code implementation ICLR 2021 Brennan Shacklett, Erik Wijmans, Aleksei Petrenko, Manolis Savva, Dhruv Batra, Vladlen Koltun, Kayvon Fatahalian

We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work, realizing end-to-end training speeds of over 19, 000 frames of experience per second on a single GPU and up to 72, 000 frames per second on a single eight-GPU machine.

PointGoal Navigation

Probabilistic two-stage detection

2 code implementations12 Mar 2021 Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl

We develop a probabilistic interpretation of two-stage object detection.

Ranked #16 on Object Detection on COCO test-dev (using extra training data)

Object Detection Region Proposal

Self-supervised Geometric Perception

2 code implementations CVPR 2021 Heng Yang, Wei Dong, Luca Carlone, Vladlen Koltun

We present self-supervised geometric perception (SGP), the first general framework to learn a feature descriptor for correspondence matching without any ground-truth geometric model labels (e. g., camera poses, rigid transformations).

Point Cloud Registration Pose Estimation

Learning a unified label space

no code implementations1 Jan 2021 Xingyi Zhou, Vladlen Koltun, Philipp Kraehenbuehl

These labels span many diverse datasets with potentially inconsistent semantic labels.

Instance Segmentation Object Detection +1

Point Transformer

6 code implementations ICCV 2021 Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, Vladlen Koltun

For example, on the challenging S3DIS dataset for large-scale semantic scene segmentation, the Point Transformer attains an mIoU of 70. 4% on Area 5, outperforming the strongest prior model by 3. 3 absolute percentage points and crossing the 70% mIoU threshold for the first time.

3D Part Segmentation 3D Point Cloud Classification +4

Stable View Synthesis

1 code implementation CVPR 2021 Gernot Riegler, Vladlen Koltun

The core of SVS is view-dependent on-surface feature aggregation, in which directional feature vectors at each 3D point are processed to produce a new feature vector for a ray that maps this point into the new target view.

Learning Quadrupedal Locomotion over Challenging Terrain

1 code implementation21 Oct 2020 Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter

The trained controller has taken two generations of quadrupedal ANYmal robots to a variety of natural environments that are beyond the reach of prior published work in legged locomotion.

NeRF++: Analyzing and Improving Neural Radiance Fields

3 code implementations15 Oct 2020 Kai Zhang, Gernot Riegler, Noah Snavely, Vladlen Koltun

Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings, including 360 capture of bounded scenes and forward-facing capture of bounded and unbounded scenes.

OpenBot: Turning Smartphones into Robots

1 code implementation24 Aug 2020 Matthias Müller, Vladlen Koltun

We develop a software stack that allows smartphones to use this body for mobile operation and demonstrate that the system is sufficiently powerful to support advanced robotics workloads such as person following and real-time autonomous navigation in unstructured environments.

Autonomous Navigation

Free View Synthesis

1 code implementation ECCV 2020 Gernot Riegler, Vladlen Koltun

We present a method for novel view synthesis from input images that are freely distributed around a scene.

Novel View Synthesis

Drinking from a Firehose: Continual Learning with Web-scale Natural Language

1 code implementation18 Jul 2020 Hexiang Hu, Ozan Sener, Fei Sha, Vladlen Koltun

Collectively, the POLL problem setting, the Firehose datasets, and the ConGraD algorithm enable a complete benchmark for reproducible research on web-scale continual learning.

Continual Learning

Scaling Imitation Learning in Minecraft

1 code implementation6 Jul 2020 Artemij Amiranashvili, Nicolai Dorka, Wolfram Burgard, Vladlen Koltun, Thomas Brox

Imitation learning is a powerful family of techniques for learning sensorimotor coordination in immersive environments.

Data Augmentation Imitation Learning

Scalable Differentiable Physics for Learning and Control

3 code implementations ICML 2020 Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, Ming C. Lin

Differentiable physics is a powerful approach to learning and control problems that involve physical objects and environments.

Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning

1 code implementation ICML 2020 Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, Vladlen Koltun

In this work we aim to solve this problem by optimizing the efficiency and resource utilization of reinforcement learning algorithms instead of relying on distributed computation.

FPS Games General Reinforcement Learning +1

Multiscale Deep Equilibrium Models

4 code implementations NeurIPS 2020 Shaojie Bai, Vladlen Koltun, J. Zico Kolter

These simultaneously-learned multi-resolution features allow us to train a single model on a diverse set of tasks and loss functions, such as using a single MDEQ to perform both image classification and semantic segmentation.

General Classification Image Classification +1

Deep Drone Acrobatics

1 code implementation10 Jun 2020 Elia Kaufmann, Antonio Loquercio, René Ranftl, Matthias Müller, Vladlen Koltun, Davide Scaramuzza

In this paper, we propose to learn a sensorimotor policy that enables an autonomous quadrotor to fly extreme acrobatic maneuvers with only onboard sensing and computation.


Auto-decoding Graphs

no code implementations4 Jun 2020 Sohil Atul Shah, Vladlen Koltun

Graph-based normalizing flows are used to sample latent codes from the distribution learned by the auto-decoder.

High-dimensional Convolutional Networks for Geometric Pattern Recognition

3 code implementations CVPR 2020 Christopher Choy, Junha Lee, Rene Ranftl, Jaesik Park, Vladlen Koltun

Many problems in science and engineering can be formulated in terms of geometric patterns in high-dimensional spaces.

Lagrangian Fluid Simulation with Continuous Convolutions

no code implementations ICLR 2020 Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, Vladlen Koltun

We present an approach to Lagrangian fluid simulation with a new type of convolutional network.

Exploring Self-attention for Image Recognition

1 code implementation CVPR 2020 Hengshuang Zhao, Jiaya Jia, Vladlen Koltun

Recent work has shown that self-attention can serve as a basic building block for image recognition models.

Learning to Guide Random Search

1 code implementation ICLR 2020 Ozan Sener, Vladlen Koltun

In other words, we jointly learn the manifold and optimize the function.

Continuous Control

Deep Global Registration

2 code implementations CVPR 2020 Christopher Choy, Wei Dong, Vladlen Koltun

We present Deep Global Registration, a differentiable framework for pairwise registration of real-world 3D scans.

Pose Estimation

Tracking Objects as Points

5 code implementations ECCV 2020 Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl

Nowadays, tracking is dominated by pipelines that perform object detection followed by temporal association, also known as tracking-by-detection.

Multi-Object Tracking Multiple Object Tracking +1

Learning to Control PDEs with Differentiable Physics

1 code implementation ICLR 2020 Philipp Holl, Vladlen Koltun, Nils Thuerey

Predicting outcomes and planning interactions with the physical world are long-standing goals for machine learning.

Differentiable Cloth Simulation for Inverse Problems

1 code implementation NeurIPS 2019 Junbang Liang, Ming Lin, Vladlen Koltun

We propose a differentiable cloth simulator that can be embedded as a layer in deep neural networks.

Fully Convolutional Geometric Features

1 code implementation International Conference on Computer vision 2019 Christopher Choy, Jaesik Park, Vladlen Koltun

Extracting geometric features from 3D scans or point clouds is the first step in applications such as registration, reconstruction, and tracking.

3D Feature Matching 3D Point Cloud Matching +3

Consensus Maximization Tree Search Revisited

1 code implementation ICCV 2019 Zhipeng Cai, Tat-Jun Chin, Vladlen Koltun

First, we show that the consensus maximization tree structure used previously actually contains paths that connect nodes at both adjacent and non-adjacent levels.

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer

9 code implementations2 Jul 2019 René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, Vladlen Koltun

In particular, we propose a robust training objective that is invariant to changes in depth range and scale, advocate the use of principled multi-objective learning to combine data from different sources, and highlight the importance of pretraining encoders on auxiliary tasks.

Monocular Depth Estimation

The Limited Multi-Label Projection Layer

1 code implementation20 Jun 2019 Brandon Amos, Vladlen Koltun, J. Zico Kolter

We propose the Limited Multi-Label (LML) projection layer as a new primitive operation for end-to-end learning systems.

General Classification Graph Generation +1

High Speed and High Dynamic Range Video with an Event Camera

1 code implementation15 Jun 2019 Henri Rebecq, René Ranftl, Vladlen Koltun, Davide Scaramuzza

In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors.

Does computer vision matter for action?

no code implementations30 May 2019 Brady Zhou, Philipp Krähenbühl, Vladlen Koltun

Thus the central question of our work: Does computer vision matter for action?

Zoom To Learn, Learn To Zoom

1 code implementation13 May 2019 Xuaner Cecilia Zhang, Qifeng Chen, Ren Ng, Vladlen Koltun

We show how to obtain the ground-truth data with optically zoomed images and contribute a dataset, SR-RAW, for real-world computational zoom.


Deep Layers as Stochastic Solvers

no code implementations ICLR 2019 Adel Bibi, Bernard Ghanem, Vladlen Koltun, Rene Ranftl

In particular, we show that a forward pass through a standard dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex optimization objective with a single iteration of a $\tau$-nice Proximal Stochastic Gradient method.

Events-to-Video: Bringing Modern Computer Vision to Event Cameras

1 code implementation CVPR 2019 Henri Rebecq, René Ranftl, Vladlen Koltun, Davide Scaramuzza

Since the output of event cameras is fundamentally different from conventional cameras, it is commonly accepted that they require the development of specialized algorithms to accommodate the particular nature of events.

Benchmarking Classic and Learned Navigation in Complex 3D Environments

1 code implementation30 Jan 2019 Dmytro Mishkin, Alexey Dosovitskiy, Vladlen Koltun

However, this new line of work is largely disconnected from well-established classic navigation approaches.

Learning agile and dynamic motor skills for legged robots

1 code implementation24 Jan 2019 Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, Marco Hutter

In the present work, we introduce a method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system, thereby leveraging fast, automated, and cost-effective data generation schemes.

Legged Robots

Motion Perception in Reinforcement Learning with Dynamic Objects

no code implementations10 Jan 2019 Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, Thomas Brox

In dynamic environments, learned controllers are supposed to take motion into account when selecting the action to be taken.

Continuous Control

Trellis Networks for Sequence Modeling

1 code implementation ICLR 2019 Shaojie Bai, J. Zico Kolter, Vladlen Koltun

On the other hand, we show that truncated recurrent networks are equivalent to trellis networks with special sparsity structure in their weight matrices.

Language Modelling Sequential Image Classification

Multi-Task Learning as Multi-Objective Optimization

4 code implementations NeurIPS 2018 Ozan Sener, Vladlen Koltun

These algorithms are not directly applicable to large-scale learning problems since they scale poorly with the dimensionality of the gradients and the number of tasks.

Depth Estimation General Classification +5

On Offline Evaluation of Vision-based Driving Models

1 code implementation ECCV 2018 Felipe Codevilla, Antonio M. López, Vladlen Koltun, Alexey Dosovitskiy

We show that the correlation of offline evaluation with driving quality can be significantly improved by selecting an appropriate validation dataset and suitable offline metrics.

Autonomous Driving

Deep Fundamental Matrix Estimation

no code implementations ECCV 2018 Rene Ranftl, Vladlen Koltun

We present an approach to robust estimation of fundamental matrices from noisy data contaminated by outliers.

On Evaluation of Embodied Navigation Agents

9 code implementations18 Jul 2018 Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, Amir R. Zamir

Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence.

Speech Denoising with Deep Feature Losses

4 code implementations27 Jun 2018 Francois G. Germain, Qifeng Chen, Vladlen Koltun

We present an end-to-end deep learning approach to denoising speech signals by processing the raw waveform directly.

Audio Tagging Denoising +3

TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning

1 code implementation ICLR 2018 Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, Thomas Brox

Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators.

Driving Policy Transfer via Modularity and Abstraction

no code implementations25 Apr 2018 Matthias Müller, Alexey Dosovitskiy, Bernard Ghanem, Vladlen Koltun

Simulation can help end-to-end driving systems by providing a cheap, safe, and diverse training environment.

Autonomous Driving

Deep Continuous Clustering

2 code implementations ICLR 2018 Sohil Atul Shah, Vladlen Koltun

We present a clustering algorithm that performs nonlinear dimensionality reduction and clustering jointly.

Dimensionality Reduction

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

26 code implementations4 Mar 2018 Shaojie Bai, J. Zico Kolter, Vladlen Koltun

Our results indicate that a simple convolutional architecture outperforms canonical recurrent networks such as LSTMs across a diverse range of tasks and datasets, while demonstrating longer effective memory.

Language Modelling Machine Translation +3

Semi-parametric Topological Memory for Navigation

1 code implementation ICLR 2018 Nikolay Savinov, Alexey Dosovitskiy, Vladlen Koltun

We introduce a new memory architecture for navigation in previously unseen environments, inspired by landmark-based navigation in animals.

Open3D: A Modern Library for 3D Data Processing

14 code implementations30 Jan 2018 Qian-Yi Zhou, Jaesik Park, Vladlen Koltun

The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python.

MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments

2 code implementations11 Dec 2017 Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, Vladlen Koltun

We present MINOS, a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments.

Learning Compact Geometric Features

1 code implementation ICCV 2017 Marc Khoury, Qian-Yi Zhou, Vladlen Koltun

We present an approach to learning features that represent the local geometry around a point in an unstructured point cloud.

Point Cloud Registration

Fast Image Processing with Fully-Convolutional Networks

2 code implementations ICCV 2017 Qifeng Chen, Jia Xu, Vladlen Koltun

Our approach uses a fully-convolutional network that is trained on input-output pairs that demonstrate the operator's action.

Style Transfer

Dilated Residual Networks

2 code implementations CVPR 2017 Fisher Yu, Vladlen Koltun, Thomas Funkhouser

Convolutional networks for image classification progressively reduce resolution until the image is represented by tiny feature maps in which the spatial structure of the scene is no longer discernible.

General Classification Image Classification +3

Learning to Act by Predicting the Future

2 code implementations6 Nov 2016 Alexey Dosovitskiy, Vladlen Koltun

A model trained using the presented approach won the Full Deathmatch track of the Visual Doom AI Competition, which was held in previously unseen environments.

Playing for Data: Ground Truth from Computer Games

1 code implementation7 Aug 2016 Stephan R. Richter, Vibhav Vineet, Stefan Roth, Vladlen Koltun

Recent progress in computer vision has been driven by high-capacity models trained on large datasets.

Semantic Segmentation

Direct Sparse Odometry

2 code implementations9 Jul 2016 Jakob Engel, Vladlen Koltun, Daniel Cremers

We propose a novel direct sparse visual odometry formulation.

Visual Odometry

Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids

no code implementations CVPR 2016 Qifeng Chen, Vladlen Koltun

The approach optimizes a classical optical flow objective over the full space of mappings between discrete grids.

Optical Flow Estimation

A Large Dataset of Object Scans

1 code implementation8 Feb 2016 Sungjoon Choi, Qian-Yi Zhou, Stephen Miller, Vladlen Koltun

We have created a dataset of more than ten thousand 3D scans of real objects.

Multi-Scale Context Aggregation by Dilated Convolutions

9 code implementations23 Nov 2015 Fisher Yu, Vladlen Koltun

State-of-the-art models for semantic segmentation are based on adaptations of convolutional networks that had originally been designed for image classification.

General Classification Real-Time Semantic Segmentation

Depth Camera Tracking With Contour Cues

no code implementations CVPR 2015 Qian-Yi Zhou, Vladlen Koltun

We present an approach for tracking camera pose in real time given a stream of depth images.

Pose Estimation

Simultaneous Localization and Calibration: Self-Calibration of Consumer Depth Cameras

no code implementations CVPR 2014 Qian-Yi Zhou, Vladlen Koltun

We describe an approach for simultaneous localization and calibration of a stream of range images.

Fast MRF Optimization with Application to Depth Reconstruction

no code implementations CVPR 2014 Qifeng Chen, Vladlen Koltun

We describe a simple and fast algorithm for optimizing Markov random fields over images.

Variational Policy Search via Trajectory Optimization

no code implementations NeurIPS 2013 Sergey Levine, Vladlen Koltun

In order to learn effective control policies for dynamical systems, policy search methods must be able to discover successful executions of the desired task.

Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

3 code implementations20 Oct 2012 Philipp Krähenbühl, Vladlen Koltun

In this paper, we consider fully connected CRF models defined on the complete set of pixels in an image.

Semantic Segmentation

Feature Construction for Inverse Reinforcement Learning

no code implementations NeurIPS 2010 Sergey Levine, Zoran Popovic, Vladlen Koltun

The goal of inverse reinforcement learning is to find a reward function for a Markov decision process, given example traces from its optimal policy.

Cannot find the paper you are looking for? You can Submit a new open access paper.