Search Results for author: Raquel Urtasun

Found 179 papers, 38 papers with code

NP-DRAW: A Non-Parametric Structured Latent Variable Model for Image Generation

1 code implementation25 Jun 2021 Xiaohui Zeng, Raquel Urtasun, Richard Zemel, Sanja Fidler, Renjie Liao

1) We propose a non-parametric prior distribution over the appearance of image parts so that the latent variable ``what-to-draw'' per step becomes a categorical random variable.

Image Generation

Just Label What You Need: Fine-Grained Active Selection for Perception and Prediction through Partially Labeled Scenes

no code implementations8 Apr 2021 Sean Segal, Nishanth Kumar, Sergio Casas, Wenyuan Zeng, Mengye Ren, Jingkang Wang, Raquel Urtasun

As data collection is often significantly cheaper than labeling in this domain, the decision of which subset of examples to label can have a profound impact on model performance.

Active Learning

IntentNet: Learning to Predict Intention from Raw Sensor Data

no code implementations20 Jan 2021 Sergio Casas, Wenjie Luo, Raquel Urtasun

In order to plan a safe maneuver, self-driving vehicles need to understand the intent of other traffic participants.

Deep Feedback Inverse Problem Solver

no code implementations ECCV 2020 Wei-Chiu Ma, Shenlong Wang, Jiayuan Gu, Sivabalan Manivasagam, Antonio Torralba, Raquel Urtasun

Specifically, at each iteration, the neural network takes the feedback as input and outputs an update on the current estimation.

Pose Estimation

Non-parametric Memory for Spatio-Temporal Segmentation of Construction Zones for Self-Driving

no code implementations18 Jan 2021 Min Bai, Shenlong Wang, Kelvin Wong, Ersin Yumer, Raquel Urtasun

In this paper, we introduce a non-parametric memory representation for spatio-temporal segmentation that captures the local space and time around an autonomous vehicle (AV).

MP3: A Unified Model to Map, Perceive, Predict and Plan

no code implementations CVPR 2021 Sergio Casas, Abbas Sadat, Raquel Urtasun

High-definition maps (HD maps) are a key component of most modern self-driving systems due to their valuable semantic and geometric information.

Deep Structured Reactive Planning

no code implementations18 Jan 2021 Jerry Liu, Wenyuan Zeng, Raquel Urtasun, Ersin Yumer

An intelligent agent operating in the real-world must balance achieving its goal with maintaining the safety and comfort of not only itself, but also other participants within the surrounding scene.

Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild

no code implementations18 Jan 2021 Shivam Duggal, ZiHao Wang, Wei-Chiu Ma, Sivabalan Manivasagam, Justin Liang, Shenlong Wang, Raquel Urtasun

Reconstructing high-quality 3D objects from sparse, partial observations from a single view is of crucial importance for various applications in computer vision, robotics, and graphics.

3D Object Reconstruction Curriculum Learning

End-to-end Interpretable Neural Motion Planner

no code implementations CVPR 2019 Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, Raquel Urtasun

In this paper, we propose a neural motion planner (NMP) for learning to drive autonomously in complex urban scenarios that include traffic-light handling, yielding, and interactions with multiple road-users.

Deep Multi-Task Learning for Joint Localization, Perception, and Prediction

no code implementations CVPR 2021 John Phillips, Julieta Martinez, Ioan Andrei Bârsan, Sergio Casas, Abbas Sadat, Raquel Urtasun

Over the last few years, we have witnessed tremendous progress on many subtasks of autonomous driving, including perception, motion forecasting, and motion planning.

Motion Forecasting Motion Planning +1

Deep Parametric Continuous Convolutional Neural Networks

no code implementations CVPR 2018 Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, Raquel Urtasun

Standard convolutional neural networks assume a grid structured input is available and exploit discrete convolutions as their fundamental building blocks.

Ranked #8 on Semantic Segmentation on S3DIS Area5 (mAcc metric)

Motion Estimation Point Cloud Segmentation +1

Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving

no code implementations17 Jan 2021 James Tu, Huichen Li, Xinchen Yan, Mengye Ren, Yun Chen, Ming Liang, Eilyan Bitar, Ersin Yumer, Raquel Urtasun

Yet, there have been limited studies on the adversarial robustness of multi-modal models that fuse LiDAR features with image features.

Adversarial Robustness Denoising

TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors

no code implementations CVPR 2021 Simon Suo, Sebastian Regalado, Sergio Casas, Raquel Urtasun

We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.

Common Sense Reasoning Data Augmentation

Adversarial Attacks On Multi-Agent Communication

no code implementations ICCV 2021 James Tu, TsunHsuan Wang, Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, Raquel Urtasun

Growing at a fast pace, modern autonomous systems will soon be deployed at scale, opening up the possibility for cooperative multi-agent systems.

Domain Adaptation

Asynchronous Multi-View SLAM

no code implementations17 Jan 2021 Anqi Joyce Yang, Can Cui, Ioan Andrei Bârsan, Raquel Urtasun, Shenlong Wang

Existing multi-camera SLAM systems assume synchronized shutters for all cameras, which is often not the case in practice.

Sensor Modeling

S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling

no code implementations CVPR 2021 Ze Yang, Shenlong Wang, Sivabalan Manivasagam, Zeng Huang, Wei-Chiu Ma, Xinchen Yan, Ersin Yumer, Raquel Urtasun

Constructing and animating humans is an important component for building virtual worlds in a wide variety of applications such as virtual reality or robotics testing in simulation.

Auto4D: Learning to Label 4D Objects from Sequential Point Clouds

no code implementations17 Jan 2021 Bin Yang, Min Bai, Ming Liang, Wenyuan Zeng, Raquel Urtasun

The key idea is to decompose the 4D object label into two parts: the object size in 3D that's fixed through time for rigid objects, and the motion path describing the evolution of the object's pose through time.

3D Object Detection

Cost-Efficient Online Hyperparameter Optimization

no code implementations17 Jan 2021 Jingkang Wang, Mengye Ren, Ilija Bogunovic, Yuwen Xiong, Raquel Urtasun

Recent work on hyperparameters optimization (HPO) has shown the possibility of training certain hyperparameters together with regular parameters.

Hyperparameter Optimization

PLUMENet: Efficient 3D Object Detection from Stereo Images

1 code implementation17 Jan 2021 Yan Wang, Bin Yang, Rui Hu, Ming Liang, Raquel Urtasun

In this paper we propose a model that unifies these two tasks and performs them in the same metric space.

3D Object Detection 3D Object Detection From Stereo Images +1

Network Automatic Pruning: Start NAP and Take a Nap

no code implementations17 Jan 2021 Wenyuan Zeng, Yuwen Xiong, Raquel Urtasun

This process is typically time-consuming and requires expert knowledge to achieve good results.

Network Pruning

VideoClick: Video Object Segmentation with a Single Click

no code implementations16 Jan 2021 Namdar Homayounfar, Justin Liang, Wei-Chiu Ma, Raquel Urtasun

Towards this goal, in this paper we propose a bottom up approach where given a single click for each object in a video, we obtain the segmentation masks of these objects in the full video.

Semantic Segmentation Video Object Segmentation +2

AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles

no code implementations CVPR 2021 Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, Raquel Urtasun

Importantly, by simulating directly from sensor data, we obtain adversarial scenarios that are safety-critical for the full autonomy stack.

Self-Supervised Representation Learning from Flow Equivariance

no code implementations ICCV 2021 Yuwen Xiong, Mengye Ren, Wenyuan Zeng, Raquel Urtasun

Motivated by this ability, we present a new self-supervised learning representation framework that can be directly deployed on a video stream of complex scenes with many moving objects.

Instance Segmentation Object Detection +3

Diverse Complexity Measures for Dataset Curation in Self-driving

no code implementations16 Jan 2021 Abbas Sadat, Sean Segal, Sergio Casas, James Tu, Bin Yang, Raquel Urtasun, Ersin Yumer

Our experiments on a wide range of tasks and models show that the proposed curation pipeline is able to select datasets that lead to better generalization and higher performance.

Active Learning Motion Forecasting

SceneGen: Learning to Generate Realistic Traffic Scenes

no code implementations CVPR 2021 Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, Raquel Urtasun

Existing methods typically insert actors into the scene according to a set of hand-crafted heuristics and are limited in their ability to model the true complexity and diversity of real traffic scenes, thus inducing a content gap between synthesized traffic scenes versus real ones.

LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving

no code implementations ICCV 2021 Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, Raquel Urtasun

In this paper, we present LookOut, a novel autonomy system that perceives the environment, predicts a diverse set of futures of how the scene might unroll and estimates the trajectory of the SDV by optimizing a set of contingency plans over these future realizations.

Future prediction Motion Forecasting

Safety-Oriented Pedestrian Motion and Scene Occupancy Forecasting

no code implementations7 Jan 2021 Katie Luo, Sergio Casas, Renjie Liao, Xinchen Yan, Yuwen Xiong, Wenyuan Zeng, Raquel Urtasun

On two large-scale real-world datasets, nuScenes and ATG4D, we showcase that our scene-occupancy predictions are more accurate and better calibrated than those from state-of-the-art motion forecasting methods, while also matching their performance in pedestrian motion forecasting metrics.

Motion Forecasting

DAGMapper: Learning to Map by Discovering Lane Topology

no code implementations ICCV 2019 Namdar Homayounfar, Wei-Chiu Ma, Justin Liang, Xinyu Wu, Jack Fan, Raquel Urtasun

One of the fundamental challenges to scale self-driving is being able to create accurate high definition maps (HD maps) with low cost.

Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net

no code implementations CVPR 2018 Wenjie Luo, Bin Yang, Raquel Urtasun

In this paper we propose a novel deep neural network that is able to jointly reason about 3D detection, tracking and motion forecasting given data captured by a 3D sensor.

Motion Forecasting

End-to-End Deep Structured Models for Drawing Crosswalks

no code implementations ECCV 2018 Justin Liang, Raquel Urtasun

In this paper we address the problem of detecting crosswalks from LiDAR and camera imagery.

Convolutional Recurrent Network for Road Boundary Extraction

no code implementations CVPR 2019 Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Shenlong Wang, Raquel Urtasun

Creating high definition maps that contain precise information of static elements of the scene is of utmost importance for enabling self driving cars to drive safely.

Self-Driving Cars

HDNET: Exploiting HD Maps for 3D Object Detection

no code implementations21 Dec 2020 Bin Yang, Ming Liang, Raquel Urtasun

In this paper we show that High-Definition (HD) maps provide strong priors that can boost the performance and robustness of modern 3D object detectors.

3D Object Detection

Learning to Localize Using a LiDAR Intensity Map

no code implementations20 Dec 2020 Ioan Andrei Bârsan, Shenlong Wang, Andrei Pokrovsky, Raquel Urtasun

In this paper we propose a real-time, calibration-agnostic and effective localization system for self-driving cars.

Self-Driving Cars

Learning to Localize Through Compressed Binary Maps

no code implementations CVPR 2019 Xinkai Wei, Ioan Andrei Bârsan, Shenlong Wang, Julieta Martinez, Raquel Urtasun

One of the main difficulties of scaling current localization systems to large environments is the on-board storage required for the maps.

Deep Continuous Fusion for Multi-Sensor 3D Object Detection

no code implementations ECCV 2018 Ming Liang, Bin Yang, Shenlong Wang, Raquel Urtasun

In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization.

3D Object Detection

A PAC-Bayesian Approach to Generalization Bounds for Graph Neural Networks

no code implementations ICLR 2021 Renjie Liao, Raquel Urtasun, Richard Zemel

In this paper, we derive generalization bounds for the two primary classes of graph neural networks (GNNs), namely graph convolutional networks (GCNs) and message passing GNNs (MPGNNs), via a PAC-Bayesian approach.

Generalization Bounds

GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation

2 code implementations13 Dec 2020 Xiaojuan Qi, Zhengzhe Liu, Renjie Liao, Philip H. S. Torr, Raquel Urtasun, Jiaya Jia

Note that GeoNet++ is generic and can be used in other depth/normal prediction frameworks to improve the quality of 3D reconstruction and pixel-wise accuracy of depth and surface normals.

3D Reconstruction Depth Estimation

Recovering and Simulating Pedestrians in the Wild

no code implementations16 Nov 2020 Ze Yang, Siva Manivasagam, Ming Liang, Bin Yang, Wei-Chiu Ma, Raquel Urtasun

We then incorporate the reconstructed pedestrian assets bank in a realistic LiDAR simulation system by performing motion retargeting, and show that the simulated LiDAR data can be used to significantly reduce the amount of annotated real-world data required for visual perception tasks.

Data Augmentation motion retargeting

MuSCLE: Multi Sweep Compression of LiDAR using Deep Entropy Models

no code implementations NeurIPS 2020 Sourav Biswas, Jerry Liu, Kelvin Wong, Shenlong Wang, Raquel Urtasun

Our model exploits spatio-temporal relationships across multiple LiDAR sweeps to reduce the bitrate of both geometry and intensity values.

Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs

no code implementations12 Nov 2020 Sean Segal, Eric Kee, Wenjie Luo, Abbas Sadat, Ersin Yumer, Raquel Urtasun

In this paper, we tackle the problem of spatio-temporal tagging of self-driving scenes from raw sensor data.

TAG Temporal Tagging

StrObe: Streaming Object Detection from LiDAR Packets

no code implementations12 Nov 2020 Davi Frossard, Simon Suo, Sergio Casas, James Tu, Rui Hu, Raquel Urtasun

In this paper we propose StrObe, a novel approach that minimizes latency by ingesting LiDAR packets and emitting a stream of detections without waiting for the full sweep to be built.

Object Detection

Learning to Communicate and Correct Pose Errors

no code implementations10 Nov 2020 Nicholas Vadivelu, Mengye Ren, James Tu, Jingkang Wang, Raquel Urtasun

Learned communication makes multi-agent systems more effective by aggregating distributed information.

Motion Forecasting Object Detection

Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving

no code implementations2 Nov 2020 Bob Wei, Mengye Ren, Wenyuan Zeng, Ming Liang, Bin Yang, Raquel Urtasun

In this paper, we propose an end-to-end self-driving network featuring a sparse attention module that learns to automatically attend to important regions of the input.

Motion Planning

LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusion

no code implementations2 Oct 2020 Meet Shah, Zhiling Huang, Ankit Laddha, Matthew Langford, Blake Barber, Sidney Zhang, Carlos Vallespi-Gonzalez, Raquel Urtasun

In this paper, we present LiRaNet, a novel end-to-end trajectory prediction method which utilizes radar sensor information along with widely used lidar and high definition (HD) maps.

Trajectory Prediction

Weakly-supervised 3D Shape Completion in the Wild

no code implementations ECCV 2020 Jiayuan Gu, Wei-Chiu Ma, Sivabalan Manivasagam, Wenyuan Zeng, ZiHao Wang, Yuwen Xiong, Hao Su, Raquel Urtasun

3D shape completion for real data is important but challenging, since partial point clouds acquired by real-world sensors are usually sparse, noisy and unaligned.

Point Cloud Registration Pose Estimation

Conditional Entropy Coding for Efficient Video Compression

no code implementations ECCV 2020 Jerry Liu, Shenlong Wang, Wei-Chiu Ma, Meet Shah, Rui Hu, Pranaab Dhawan, Raquel Urtasun

We propose a very simple and efficient video compression framework that only focuses on modeling the conditional entropy between frames.

MS-SSIM SSIM +1

V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction

no code implementations ECCV 2020 Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, James Tu, Raquel Urtasun

In this paper, we explore the use of vehicle-to-vehicle (V2V) communication to improve the perception and motion forecasting performance of self-driving vehicles.

Motion Forecasting

DSDNet: Deep Structured self-Driving Network

no code implementations ECCV 2020 Wenyuan Zeng, Shenlong Wang, Renjie Liao, Yun Chen, Bin Yang, Raquel Urtasun

In this paper, we propose the Deep Structured self-Driving Network (DSDNet), which performs object detection, motion prediction, and motion planning with a single neural network.

Motion Planning motion prediction +1

Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations

no code implementations ECCV 2020 Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, Raquel Urtasun

In this paper we propose a novel end-to-end learnable network that performs joint perception, prediction and motion planning for self-driving vehicles and produces interpretable intermediate representations.

Motion Planning

LoCo: Local Contrastive Representation Learning

no code implementations NeurIPS 2020 Yuwen Xiong, Mengye Ren, Raquel Urtasun

Deep neural nets typically perform end-to-end backpropagation to learn the weights, a procedure that creates synchronization constraints in the weight update step across layers and is not biologically plausible.

Contrastive Learning Instance Segmentation +3

LevelSet R-CNN: A Deep Variational Method for Instance Segmentation

no code implementations30 Jul 2020 Namdar Homayounfar, Yuwen Xiong, Justin Liang, Wei-Chiu Ma, Raquel Urtasun

Obtaining precise instance segmentation masks is of high importance in many modern applications such as robotic manipulation and autonomous driving.

Autonomous Driving Instance Segmentation +1

RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects

no code implementations ECCV 2020 Bin Yang, Runsheng Guo, Ming Liang, Sergio Casas, Raquel Urtasun

We tackle the problem of exploiting Radar for perception in the context of self-driving as Radar provides complementary information to other sensors such as LiDAR or cameras in the form of Doppler velocity.

Object Detection

Learning Lane Graph Representations for Motion Forecasting

1 code implementation ECCV 2020 Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, Raquel Urtasun

We propose a motion forecasting model that exploits a novel structured map representation as well as actor-map interactions.

Motion Forecasting Trajectory Prediction

Hierarchical Verification for Adversarial Robustness

no code implementations ICML 2020 Cong Han Lim, Raquel Urtasun, Ersin Yumer

We show that, under certain conditions on the algorithm parameters, LayerCert provably reduces the number and size of the convex programs that one needs to solve compared to GeoCert.

Adversarial Robustness

Implicit Latent Variable Model for Scene-Consistent Motion Forecasting

no code implementations ECCV 2020 Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, Raquel Urtasun

In order to plan a safe maneuver an autonomous vehicle must accurately perceive its environment, and understand the interactions among traffic participants.

Motion Forecasting Motion Planning

Multi-Agent Routing Value Iteration Network

1 code implementation ICML 2020 Quinlan Sykora, Mengye Ren, Raquel Urtasun

In this paper we tackle the problem of routing multiple agents in a coordinated manner.

Autonomous Vehicles

LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World

no code implementations CVPR 2020 Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, Raquel Urtasun

We first utilize ray casting over the 3D scene and then use a deep neural network to produce deviations from the physics-based simulation, producing realistic LiDAR point clouds.

The Importance of Prior Knowledge in Precise Multimodal Prediction

no code implementations4 Jun 2020 Sergio Casas, Cole Gulino, Simon Suo, Raquel Urtasun

Towards this goal, we design a framework that leverages REINFORCE to incorporate non-differentiable priors over sample trajectories from a probabilistic model, thus optimizing the whole distribution.

Motion Forecasting Motion Planning

ShapeAdv: Generating Shape-Aware Adversarial 3D Point Clouds

no code implementations24 May 2020 Kibok Lee, Zhuoyuan Chen, Xinchen Yan, Raquel Urtasun, Ersin Yumer

Our shape-aware adversarial attacks are orthogonal to existing point cloud based attacks and shed light on the vulnerability of 3D deep neural networks.

Physically Realizable Adversarial Examples for LiDAR Object Detection

no code implementations CVPR 2020 James Tu, Mengye Ren, Siva Manivasagam, Ming Liang, Bin Yang, Richard Du, Frank Cheng, Raquel Urtasun

Modern autonomous driving systems rely heavily on deep learning models to process point cloud sensory data; meanwhile, deep models have been shown to be susceptible to adversarial attacks with visually imperceptible perturbations.

Adversarial Defense Autonomous Driving +2

Dense RepPoints: Representing Visual Objects with Dense Point Sets

2 code implementations ECCV 2020 Ze Yang, Yinghao Xu, Han Xue, Zheng Zhang, Raquel Urtasun, Li-Wei Wang, Stephen Lin, Han Hu

We present a new object representation, called Dense RepPoints, that utilizes a large set of points to describe an object at multiple levels, including both box level and pixel level.

Object Detection

PolyTransform: Deep Polygon Transformer for Instance Segmentation

no code implementations CVPR 2020 Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, Raquel Urtasun

In this paper, we propose PolyTransform, a novel instance segmentation algorithm that produces precise, geometry-preserving masks by combining the strengths of prevailing segmentation approaches and modern polygon-based methods.

 Ranked #1 on Instance Segmentation on Cityscapes test (using extra training data)

Instance Segmentation Semantic Segmentation

Identifying Unknown Instances for Autonomous Driving

no code implementations24 Oct 2019 Kelvin Wong, Shenlong Wang, Mengye Ren, Ming Liang, Raquel Urtasun

In the past few years, we have seen great progress in perception algorithms, particular through the use of deep learning.

Autonomous Driving Instance Segmentation +1

Learning to Remember from a Multi-Task Teacher

no code implementations10 Oct 2019 Yuwen Xiong, Mengye Ren, Raquel Urtasun

Recent studies on catastrophic forgetting during sequential learning typically focus on fixing the accuracy of the predictions for a previously learned task.

Meta-Learning

Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehicles

no code implementations10 Oct 2019 Abbas Sadat, Mengye Ren, Andrei Pokrovsky, Yen-Chen Lin, Ersin Yumer, Raquel Urtasun

The motion planners used in self-driving vehicles need to generate trajectories that are safe, comfortable, and obey the traffic rules.

Trajectory Planning

DSIC: Deep Stereo Image Compression

no code implementations ICCV 2019 Jerry Liu, Shenlong Wang, Raquel Urtasun

In this paper we tackle the problem of stereo image compression, and leverage the fact that the two images have overlapping fields of view to further compress the representations.

Image Compression

DeepSignals: Predicting Intent of Drivers Through Visual Signals

no code implementations3 May 2019 Davi Frossard, Eric Kee, Raquel Urtasun

Detecting the intention of drivers is an essential task in self-driving, necessary to anticipate sudden events like lane changes and stops.

Deep Rigid Instance Scene Flow

no code implementations CVPR 2019 Wei-Chiu Ma, Shenlong Wang, Rui Hu, Yuwen Xiong, Raquel Urtasun

In this paper we tackle the problem of scene flow estimation in the context of self-driving.

Scene Flow Estimation

PIXOR: Real-time 3D Object Detection from Point Clouds

1 code implementation CVPR 2018 Bin Yang, Wenjie Luo, Raquel Urtasun

Existing approaches are, however, expensive in computation due to high dimensionality of point clouds.

3D Object Detection Autonomous Driving

LanczosNet: Multi-Scale Deep Graph Convolutional Networks

1 code implementation ICLR 2019 Renjie Liao, Zhizhen Zhao, Raquel Urtasun, Richard S. Zemel

We propose the Lanczos network (LanczosNet), which uses the Lanczos algorithm to construct low rank approximations of the graph Laplacian for graph convolution.

Node Classification

SurfConv: Bridging 3D and 2D Convolution for RGBD Images

1 code implementation CVPR 2018 Hang Chu, Wei-Chiu Ma, Kaustav Kundu, Raquel Urtasun, Sanja Fidler

On the other hand, 3D convolution wastes a large amount of memory on mostly unoccupied 3D space, which consists of only the surface visible to the sensor.

3D Semantic Segmentation

Graph HyperNetworks for Neural Architecture Search

1 code implementation ICLR 2019 Chris Zhang, Mengye Ren, Raquel Urtasun

Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs.

Neural Architecture Search

MLPrune: Multi-Layer Pruning for Automated Neural Network Compression

no code implementations27 Sep 2018 Wenyuan Zeng, Raquel Urtasun

Model compression can significantly reduce the computation and memory footprint of large neural networks.

Neural Network Compression

Single Image Intrinsic Decomposition without a Single Intrinsic Image

no code implementations ECCV 2018 Wei-Chiu Ma, Hang Chu, Bolei Zhou, Raquel Urtasun, Antonio Torralba

At inference time, our model can be easily reduced to a single stream module that performs intrinsic decomposition on a single input image.

Intrinsic Image Decomposition

End-to-end Learning of Multi-sensor 3D Tracking by Detection

no code implementations29 Jun 2018 Davi Frossard, Raquel Urtasun

In this paper we propose a novel approach to tracking by detection that can exploit both cameras as well as LIDAR data to produce very accurate 3D trajectories.

Multiple Object Tracking

GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation

1 code implementation CVPR 2018 Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, Jiaya Jia

In this paper, we propose Geometric Neural Network (GeoNet) to jointly predict depth and surface normal maps from a single image.

Depth Estimation

Matching Adversarial Networks

no code implementations CVPR 2018 Gellért Máttyus, Raquel Urtasun

We argue that the main difficulty of applying CGANs to supervised tasks is that the generator training consists of optimizing a loss function that does not depend directly on the ground truth labels.

Depth Estimation Line Detection +1

Learning to Reweight Examples for Robust Deep Learning

9 code implementations ICML 2018 Mengye Ren, Wenyuan Zeng, Bin Yang, Raquel Urtasun

Deep neural networks have been shown to be very powerful modeling tools for many supervised learning tasks involving complex input patterns.

Meta-Learning

Inference in Probabilistic Graphical Models by Graph Neural Networks

1 code implementation21 Mar 2018 KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq Pitkow

Message-passing algorithms, such as belief propagation, are a natural way to disseminate evidence amongst correlated variables while exploiting the graph structure, but these algorithms can struggle when the conditional dependency graphs contain loops.

Decision Making

Learning deep structured active contours end-to-end

1 code implementation CVPR 2018 Diego Marcos, Devis Tuia, Benjamin Kellenberger, Lisa Zhang, Min Bai, Renjie Liao, Raquel Urtasun

The world is covered with millions of buildings, and precisely knowing each instance's position and extents is vital to a multitude of applications.

Instance Segmentation Semantic Segmentation

Reviving and Improving Recurrent Back-Propagation

1 code implementation ICML 2018 Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urtasun, Richard Zemel

We examine all RBP variants along with BPTT and TBPTT in three different application domains: associative memory with continuous Hopfield networks, document classification in citation networks using graph neural networks and hyperparameter optimization for fully connected networks.

Document Classification Hyperparameter Optimization

SBNet: Sparse Blocks Network for Fast Inference

2 code implementations CVPR 2018 Mengye Ren, Andrei Pokrovsky, Bin Yang, Raquel Urtasun

Conventional deep convolutional neural networks (CNNs) apply convolution operators uniformly in space across all feature maps for hundreds of layers - this incurs a high computational cost for real-time applications.

3D Object Detection Semantic Segmentation

Be Your Own Prada: Fashion Synthesis with Structural Coherence

no code implementations ICCV 2017 Shizhan Zhu, Sanja Fidler, Raquel Urtasun, Dahua Lin, Chen Change Loy

In the second stage, a generative model with a newly proposed compositional mapping layer is used to render the final image with precise regions and textures conditioned on this map.

Fashion Synthesis Semantic Segmentation

SGN: Sequential Grouping Networks for Instance Segmentation

no code implementations ICCV 2017 Shu Liu, Jiaya Jia, Sanja Fidler, Raquel Urtasun

By exploiting two-directional information, the second network groups horizontal and vertical lines into connected components.

Instance Segmentation Semantic Segmentation

DeepRoadMapper: Extracting Road Topology From Aerial Images

no code implementations ICCV 2017 Gellert Mattyus, Wenjie Luo, Raquel Urtasun

In contrast, in this paper we propose an approach that directly estimates road topology from aerial images.

Autonomous Driving

3D Graph Neural Networks for RGBD Semantic Segmentation

2 code implementations ICCV 2017 Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, Raquel Urtasun

Each node in the graph corresponds to a set of points and is associated with a hidden representation vector initialized with an appearance feature extracted by a unary CNN from 2D images.

Semantic Segmentation

Deep Spectral Clustering Learning

no code implementations ICML 2017 Marc T. Law, Raquel Urtasun, Richard S. Zemel

We derive a closed-form expression for the gradient that is efficient to compute: the complexity to compute the gradient is linear in the size of the training mini-batch and quadratic in the representation dimensionality.

Metric Learning

The Reversible Residual Network: Backpropagation Without Storing Activations

8 code implementations NeurIPS 2017 Aidan N. Gomez, Mengye Ren, Raquel Urtasun, Roger B. Grosse

Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider.

General Classification Image Classification

Sports Field Localization via Deep Structured Models

no code implementations CVPR 2017 Namdar Homayounfar, Sanja Fidler, Raquel Urtasun

In this work, we propose a novel way of efficiently localizing a sports field from a single broadcast image of the game.

Semantic Segmentation

Annotating Object Instances with a Polygon-RNN

1 code implementation CVPR 2017 Lluis Castrejon, Kaustav Kundu, Raquel Urtasun, Sanja Fidler

We show that our approach speeds up the annotation process by a factor of 4. 7 across all classes in Cityscapes, while achieving 78. 4% agreement in IoU with original ground-truth, matching the typical agreement between human annotators.

Semantic Segmentation

Towards Diverse and Natural Image Descriptions via a Conditional GAN

1 code implementation ICCV 2017 Bo Dai, Sanja Fidler, Raquel Urtasun, Dahua Lin

Despite the substantial progress in recent years, the image captioning techniques are still far from being perfect. Sentences produced by existing methods, e. g. those based on RNNs, are often overly rigid and lacking in variability.

Image Captioning

MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving

15 code implementations22 Dec 2016 Marvin Teichmann, Michael Weber, Marius Zoellner, Roberto Cipolla, Raquel Urtasun

While most approaches to semantic reasoning have focused on improving performance, in this paper we argue that computational times are very important in order to enable real time applications such as autonomous driving.

Autonomous Driving General Classification +1

TorontoCity: Seeing the World with a Million Eyes

no code implementations ICCV 2017 Shenlong Wang, Min Bai, Gellert Mattyus, Hang Chu, Wenjie Luo, Bin Yang, Justin Liang, Joel Cheverie, Sanja Fidler, Raquel Urtasun

In this paper we introduce the TorontoCity benchmark, which covers the full greater Toronto area (GTA) with 712. 5 $km^2$ of land, 8439 $km$ of road and around 400, 000 buildings.

Instance Segmentation Semantic Segmentation

Proximal Deep Structured Models

no code implementations NeurIPS 2016 Shenlong Wang, Sanja Fidler, Raquel Urtasun

Many problems in real-world applications involve predicting continuous-valued random variables that are statistically related.

Image Denoising Optical Flow Estimation

Deep Watershed Transform for Instance Segmentation

3 code implementations CVPR 2017 Min Bai, Raquel Urtasun

Most contemporary approaches to instance segmentation use complex pipelines involving conditional random fields, recurrent neural networks, object proposals, or template matching schemes.

Instance Segmentation Semantic Segmentation +1

Efficient Summarization with Read-Again and Copy Mechanism

no code implementations10 Nov 2016 Wenyuan Zeng, Wenjie Luo, Sanja Fidler, Raquel Urtasun

Towards this goal, we first introduce a simple mechanism that first reads the input sequence before committing to a representation of each word.

Find your Way by Observing the Sun and Other Semantic Cues

no code implementations23 Jun 2016 Wei-Chiu Ma, Shenlong Wang, Marcus A. Brubaker, Sanja Fidler, Raquel Urtasun

In this paper we present a robust, efficient and affordable approach to self-localization which does not require neither GPS nor knowledge about the appearance of the world.

HD Maps: Fine-Grained Road Segmentation by Parsing Ground and Aerial Images

no code implementations CVPR 2016 Gellert Mattyus, Shenlong Wang, Sanja Fidler, Raquel Urtasun

In this paper we present an approach to enhance existing maps with fine grained segmentation categories such as parking spots and sidewalk, as well as the number and location of road lanes.

Soccer Field Localization from a Single Image

no code implementations10 Apr 2016 Namdar Homayounfar, Sanja Fidler, Raquel Urtasun

In this work, we propose a novel way of efficiently localizing a soccer field from a single broadcast image of the game.

Exploiting Semantic Information and Deep Matching for Optical Flow

no code implementations6 Apr 2016 Min Bai, Wenjie Luo, Kaustav Kundu, Raquel Urtasun

We tackle the problem of estimating optical flow from a monocular camera in the context of autonomous driving.

Autonomous Driving Optical Flow Estimation

Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs

no code implementations CVPR 2016 Ziyu Zhang, Sanja Fidler, Raquel Urtasun

Our aim is to provide a pixel-wise instance-level labeling of a monocular image in the context of autonomous driving.

Autonomous Driving

Lost Shopping! Monocular Localization in Large Indoor Spaces

no code implementations ICCV 2015 Shenlong Wang, Sanja Fidler, Raquel Urtasun

In this paper we propose a novel approach to localization in very large indoor spaces (i. e., 200+ store shopping malls) that takes a single image and a floor plan of the environment as input.

Translation

Enhancing Road Maps by Parsing Aerial Images Around the World

no code implementations ICCV 2015 Gellert Mattyus, Shenlong Wang, Sanja Fidler, Raquel Urtasun

In recent years, contextual models that exploit maps have been shown to be very effective for many recognition and localization tasks.

Semantic Segmentation

Order-Embeddings of Images and Language

2 code implementations19 Nov 2015 Ivan Vendrov, Ryan Kiros, Sanja Fidler, Raquel Urtasun

Hypernymy, textual entailment, and image captioning can be seen as special cases of a single visual-semantic hierarchy over words, sentences, and images.

Cross-Modal Retrieval Image Captioning +1

Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets

no code implementations17 Nov 2015 Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor Aamodt, Natalie Enright Jerger, Raquel Urtasun, Andreas Moshovos

A diverse set of CNNs is analyzed showing that compared to a conventional implementation using a 32-bit floating-point representation for all layers, and with less than 1% loss in relative accuracy, the data footprint required by these networks can be reduced by an average of 74% and up to 92%.

Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

3 code implementations ICCV 2015 Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, Sanja Fidler

Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story.

Sentence Embedding

Skip-Thought Vectors

16 code implementations NeurIPS 2015 Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, Sanja Fidler

The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice.

Learning to Segment Under Various Forms of Weak Supervision

no code implementations CVPR 2015 Jia Xu, Alexander G. Schwing, Raquel Urtasun

Despite the promising performance of conventional fully supervised algorithms, semantic segmentation has remained an important, yet challenging task.

Semantic Segmentation

Rent3D: Floor-Plan Priors for Monocular Layout Estimation

no code implementations CVPR 2015 Chenxi Liu, Alexander G. Schwing, Kaustav Kundu, Raquel Urtasun, Sanja Fidler

What sets us apart from past work in layout estimation is the use of floor plans as a source of prior knowledge, as well as localization of each image within a bigger space (apartment).

Fully Connected Deep Structured Networks

no code implementations9 Mar 2015 Alexander G. Schwing, Raquel Urtasun

Convolutional neural networks with many layers have recently been shown to achieve excellent results on many high-level tasks such as image classification, object detection and more recently also semantic segmentation.

General Classification Image Classification +2

Generating Multi-Sentence Lingual Descriptions of Indoor Scenes

no code implementations28 Feb 2015 Dahua Lin, Chen Kong, Sanja Fidler, Raquel Urtasun

This paper proposes a novel framework for generating lingual descriptions of indoor scenes.

Text Generation

Neuroaesthetics in Fashion: Modeling the Perception of Fashionability

no code implementations Conference 2015 Edgar Simo-Serra, Sanja Fidler, Francesc Moreno-Noguer, Raquel Urtasun

Importantly, our model is able to give rich feedback back to the user, conveying which garments or even scenery she/he should change in order to improve fashionability.

Efficient Inference of Continuous Markov Random Fields with Polynomial Potentials

no code implementations NeurIPS 2014 Shenlong Wang, Alex Schwing, Raquel Urtasun

In this paper, we prove that every multivariate polynomial with even degree can be decomposed into a sum of convex and concave polynomials.

3D Reconstruction Image Denoising

Message Passing Inference for Large Scale Graphical Models with High Order Potentials

no code implementations NeurIPS 2014 Jian Zhang, Alex Schwing, Raquel Urtasun

To keep up with the Big Data challenge, parallelized algorithms based on dual decomposition have been proposed to perform inference in Markov random fields.

Semantic Segmentation

Learning Deep Structured Models

no code implementations9 Jul 2014 Liang-Chieh Chen, Alexander G. Schwing, Alan L. Yuille, Raquel Urtasun

Towards this goal, we propose a training algorithm that is able to learn structured models jointly with deep features that form the MRF potentials.

Multi-class Classification

Human-Machine CRFs for Identifying Bottlenecks in Holistic Scene Understanding

no code implementations16 Jun 2014 Roozbeh Mottaghi, Sanja Fidler, Alan Yuille, Raquel Urtasun, Devi Parikh

Recent trends in image understanding have pushed for holistic scene understanding models that jointly reason about various tasks such as object detection, scene recognition, shape analysis, contextual reasoning, and local appearance based classifiers.

Object Detection Scene Recognition +2

Tell Me What You See and I will Show You Where It Is

no code implementations CVPR 2014 Jia Xu, Alexander G. Schwing, Raquel Urtasun

We tackle the problem of weakly labeled semantic segmentation, where the only source of annotation are image tags encoding which classes are present in the scene.

Semantic Segmentation Structured Prediction +1

Beat the MTurkers: Automatic Image Labeling from Weak 3D Supervision

1 code implementation CVPR 2014 Liang-Chieh Chen, Sanja Fidler, Alan L. Yuille, Raquel Urtasun

Labeling large-scale datasets with very accurate object segmentations is an elaborate task that requires a high degree of quality control and a budget of tens or hundreds of thousands of dollars.

Autonomous Driving

Latent Structured Active Learning

no code implementations NeurIPS 2013 Wenjie Luo, Alex Schwing, Raquel Urtasun

In this paper we present active learning algorithms in the context of structured prediction problems.

Active Learning Structured Prediction

Bottom-Up Segmentation for Top-Down Detection

no code implementations CVPR 2013 Sanja Fidler, Roozbeh Mottaghi, Alan Yuille, Raquel Urtasun

When employing the parts, we outperform the original DPM [14] in 19 out of 20 classes, achieving an improvement of 8% AP.

Object Detection Semantic Segmentation

A Sentence Is Worth a Thousand Pixels

no code implementations CVPR 2013 Sanja Fidler, Abhishek Sharma, Raquel Urtasun

We are interested in holistic scene understanding where images are accompanied with text in the form of complex sentential descriptions.

Re-Ranking Scene Understanding +2

Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs

no code implementations CVPR 2013 Roozbeh Mottaghi, Sanja Fidler, Jian Yao, Raquel Urtasun, Devi Parikh

Recent trends in semantic image segmentation have pushed for holistic scene understanding models that jointly reason about various tasks such as object detection, scene recognition, shape analysis, contextual reasoning.

Object Detection Scene Recognition +2

Lost! Leveraging the Crowd for Probabilistic Visual Self-Localization

no code implementations CVPR 2013 Marcus A. Brubaker, Andreas Geiger, Raquel Urtasun

In this paper we propose an affordable solution to selflocalization, which utilizes visual odometry and road maps as the only inputs.

Visual Odometry

Robust Monocular Epipolar Flow Estimation

no code implementations CVPR 2013 Koichiro Yamaguchi, David Mcallester, Raquel Urtasun

We consider the problem of computing optical flow in monocular video taken from a moving vehicle.

Optical Flow Estimation

3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model

no code implementations NeurIPS 2012 Sanja Fidler, Sven Dickinson, Raquel Urtasun

We demonstrate the effectiveness of our approach in indoor and outdoor scenarios, and show that our approach outperforms the state-of-the-art in both 2D[Felz09] and 3D object detection[Hedau12].

3D Object Detection Viewpoint Estimation

Globally Convergent Dual MAP LP Relaxation Solvers using Fenchel-Young Margins

no code implementations NeurIPS 2012 Alex Schwing, Tamir Hazan, Marc Pollefeys, Raquel Urtasun

While finding the exact solution for the MAP inference problem is intractable for many real-world tasks, MAP LP relaxations have been shown to be very effective in practice.

Blending Learning and Inference in Structured Prediction

no code implementations8 Oct 2012 Tamir Hazan, Alexander Schwing, David Mcallester, Raquel Urtasun

In this paper we derive an efficient algorithm to learn the parameters of structured predictors in general graphical models.

Scene Understanding Semantic Segmentation +1

Joint 3D Estimation of Objects and Scene Layout

no code implementations NeurIPS 2011 Andreas Geiger, Christian Wojek, Raquel Urtasun

We propose a novel generative model that is able to reason jointly about the 3D scene layout as well as the 3D location and orientation of objects in the scene.

Learning Probabilistic Non-Linear Latent Variable Models for Tracking Complex Activities

no code implementations NeurIPS 2011 Angela Yao, Juergen Gall, Luc V. Gool, Raquel Urtasun

A common approach for handling the complexity and inherent ambiguities of 3D human pose estimation is to use pose priors learned from training data.

3D Human Pose Estimation

Sparse Coding for Learning Interpretable Spatio-Temporal Primitives

no code implementations NeurIPS 2010 Taehwan Kim, Gregory Shakhnarovich, Raquel Urtasun

Sparse coding has recently become a popular approach in computer vision to learn dictionaries of natural images.

A Primal-Dual Message-Passing Algorithm for Approximated Large Scale Structured Prediction

no code implementations NeurIPS 2010 Tamir Hazan, Raquel Urtasun

We then propose an intuitive approximation for structured prediction problems using Fenchel duality based on a local entropy approximation that computes the exact gradients of the approximated problem and is guaranteed to converge.

Image Denoising Structured Prediction

Cannot find the paper you are looking for? You can Submit a new open access paper.