Search Results for author: Silvio Savarese

Found 137 papers, 48 papers with code

Long Document Summarization with Top-down and Bottom-up Inference

no code implementations15 Mar 2022 Bo Pang, Erik Nijkamp, Wojciech Kryściński, Silvio Savarese, Yingbo Zhou, Caiming Xiong

Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.

ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation

no code implementations14 Mar 2022 Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J. Guibas, Silvio Savarese, Anima Anandkumar, Yuke Zhu

Manipulating volumetric deformable objects in the real world, like plush toys and pizza dough, bring substantial challenges due to infinite shape variations, non-rigid motions, and partial observability.

Contrastive Learning Deformable Object Manipulation

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

no code implementations9 Dec 2021 Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, Roberto Martín-Martín

Doing this is challenging for two reasons: on the data side, current interfaces make collecting high-quality human demonstrations difficult, and on the learning side, policies trained on limited data can suffer from covariate shift when deployed.

Imitation Learning

Local Calibration: Metrics and Recalibration

no code implementations29 Sep 2021 Rachel Luo, Aadyot Bhatnagar, Yu Bai, Shengjia Zhao, Huan Wang, Caiming Xiong, Silvio Savarese, Stefano Ermon, Edward Schmerling, Marco Pavone

In this work, we propose the local calibration error (LCE) to span the gap between average and individual reliability.

Decision Making Fairness

Long Document Summarization with Top-Down and Bottom-Up Representation Inference

no code implementations29 Sep 2021 Bo Pang, Erik Nijkamp, Wojciech Maciej Kryscinski, Silvio Savarese, Yingbo Zhou, Caiming Xiong

Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents.

Document Summarization

Sample-Efficient Safety Assurances using Conformal Prediction

no code implementations28 Sep 2021 Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, Marco Pavone

When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial.

Robotic Grasping

Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation

no code implementations2 Sep 2021 Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, Chelsea Finn

However, goal images also have a number of drawbacks: they are inconvenient for humans to provide, they can over-specify the desired behavior leading to a sparse reward signal, or under-specify task information in the case of non-goal reaching tasks.

Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration

1 code implementation13 Aug 2021 Chen Wang, Claudia Pérez-D'Arpino, Danfei Xu, Li Fei-Fei, C. Karen Liu, Silvio Savarese

Our method co-optimizes a human policy and a robot policy in an interactive learning process: the human policy learns to generate diverse and plausible collaborative behaviors from demonstrations while the robot policy learns to assist by estimating the unobserved latent strategy of its human collaborator.

BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments

no code implementations6 Aug 2021 Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, Li Fei-Fei

We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation.

iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks

1 code implementation6 Aug 2021 Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, Silvio Savarese

We evaluate the new capabilities of iGibson 2. 0 to enable robot learning of novel tasks, in the hope of demonstrating the potential of this new simulator to support new research in embodied AI.

Imitation Learning

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

1 code implementation6 Aug 2021 Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martín-Martín

Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation.

Imitation Learning reinforcement-learning

Discovering Generalizable Skills via Automated Generation of Diverse Tasks

no code implementations26 Jun 2021 Kuan Fang, Yuke Zhu, Silvio Savarese, Li Fei-Fei

To encourage generalizable skills to emerge, our method trains each skill to specialize in the paired task and maximizes the diversity of the generated tasks.

Hierarchical Reinforcement Learning reinforcement-learning

JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection

no code implementations16 Jun 2021 Mahsa Ehsanpour, Fatemeh Saleh, Silvio Savarese, Ian Reid, Hamid Rezatofighi

However, learning to recognise human actions and their social interactions in an unconstrained real-world environment comprising numerous people, with potentially highly unbalanced and long-tailed distributed action labels from a stream of sensory data captured from a mobile robot platform remains a significant challenge, not least owing to the lack of a reflective large-scale dataset.

Action Detection Action Understanding +1

TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild

no code implementations ICCV 2021 Vida Adeli, Mahsa Ehsanpour, Ian Reid, Juan Carlos Niebles, Silvio Savarese, Ehsan Adeli, Hamid Rezatofighi

Joint forecasting of human trajectory and pose dynamics is a fundamental building block of various applications ranging from robotics and autonomous driving to surveillance systems.

Autonomous Driving Frame +1

LASER: Learning a Latent Action Space for Efficient Reinforcement Learning

no code implementations29 Mar 2021 Arthur Allshire, Roberto Martín-Martín, Charles Lin, Shawn Manuel, Silvio Savarese, Animesh Garg

Additionally, similar tasks or instances of the same task family impose latent manifold constraints on the most effective action space: the task family can be best solved with actions in a manifold of the entire action space of the robot.


Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control

no code implementations28 Feb 2021 Chen Wang, Rui Wang, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, Danfei Xu

Key to such capability is hand-eye coordination, a cognitive ability that enables humans to adaptively direct their movements at task-relevant objects and be invariant to the objects' absolute spatial location.

Imitation Learning

Localized Calibration: Metrics and Recalibration

no code implementations22 Feb 2021 Rachel Luo, Aadyot Bhatnagar, Huan Wang, Caiming Xiong, Silvio Savarese, Yu Bai, Shengjia Zhao, Stefano Ermon

Probabilistic classifiers output confidence scores along with their predictions, and these confidence scores must be well-calibrated (i. e. reflect the true probability of an event) to be meaningful and useful for downstream tasks.

Decision Making

Embodied Intelligence via Learning and Evolution

1 code implementation3 Feb 2021 Agrim Gupta, Silvio Savarese, Surya Ganguli, Li Fei-Fei

However, the principles governing relations between environmental complexity, evolved morphology, and the learnability of intelligent control, remain elusive, partially due to the substantial challenge of performing large-scale in silico experiments on evolution and learning.

Learning Multi-Arm Manipulation Through Collaborative Teleoperation

no code implementations12 Dec 2020 Albert Tung, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, Silvio Savarese

To address these challenges, we present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks.

Imitation Learning

Human-in-the-Loop Imitation Learning using Remote Teleoperation

no code implementations12 Dec 2020 Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, Silvio Savarese

We develop a simple and effective algorithm to train the policy iteratively on new data collected by the system that encourages the policy to learn how to traverse bottlenecks through the interventions.

Imitation Learning

Topological Planning with Transformers for Vision-and-Language Navigation

no code implementations CVPR 2021 Kevin Chen, Junshen K. Chen, Jo Chuang, Marynel Vázquez, Silvio Savarese

Conventional approaches to vision-and-language navigation (VLN) are trained end-to-end but struggle to perform well in freely traversable environments.

Vision and Language Navigation

Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search

no code implementations7 Dec 2020 Andrey Kurenkov, Roberto Martín-Martín, Jeff Ichnowski, Ken Goldberg, Silvio Savarese

We propose to use a 3D scene graph representation to capture the hierarchical, semantic, and geometric aspects of this problem.

Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning

2 code implementations16 Oct 2020 Claudia Pérez-D'Arpino, Can Liu, Patrick Goebel, Roberto Martín-Martín, Silvio Savarese

Navigating fluently around pedestrians is a necessary capability for mobile robots deployed in human environments, such as buildings and homes.

reinforcement-learning Robot Navigation

Privacy Preserving Recalibration under Domain Shift

no code implementations21 Aug 2020 Rachel Luo, Shengjia Zhao, Jiaming Song, Jonathan Kuck, Stefano Ermon, Silvio Savarese

In an extensive empirical study, we find that our algorithm improves calibration on domain-shift benchmarks under the constraints of differential privacy.

ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation

no code implementations18 Aug 2020 Fei Xia, Chengshu Li, Roberto Martín-Martín, Or Litany, Alexander Toshev, Silvio Savarese

To validate our method, we apply ReLMoGen to two types of tasks: 1) Interactive Navigation tasks, navigation problems where interactions with the environment are required to reach the destination, and 2) Mobile Manipulation tasks, manipulation tasks that require moving the robot base.

Continuous Control Hierarchical Reinforcement Learning +1

Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter

no code implementations13 Aug 2020 Andrey Kurenkov, Joseph Taglic, Rohun Kulkarni, Marcus Dominguez-Kuhne, Animesh Garg, Roberto Martín-Martín, Silvio Savarese

When searching for objects in cluttered environments, it is often necessary to perform complex interactions in order to move occluding objects out of the way and fully reveal the object of interest and make it graspable.

How Trustworthy are Performance Evaluations for Basic Vision Tasks?

no code implementations8 Aug 2020 Hamid Rezatofighi, Tran Thien Dat Nguyen, Ba-Ngu Vo, Ba-Tuong Vo, Silvio Savarese, Ian Reid

This paper examines performance evaluation criteria for basic vision tasks involving sets of objects namely, object detection, instance-level segmentation and multi-object tracking.

Multi-Object Tracking Object Detection

Goal-Aware Prediction: Learning to Model What Matters

no code implementations ICML 2020 Suraj Nair, Silvio Savarese, Chelsea Finn

In this paper, we propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space, resulting in a learning objective that more closely matches the downstream task.

Adaptive Procedural Task Generation for Hard-Exploration Problems

no code implementations ICLR 2021 Kuan Fang, Yuke Zhu, Silvio Savarese, Li Fei-Fei

To enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balancing the agent's performance in the generated tasks and the similarity to the target tasks.

Generative Sparse Detection Networks for 3D Single-shot Object Detection

3 code implementations ECCV 2020 JunYoung Gwak, Christopher Choy, Silvio Savarese

3D object detection has been widely studied due to its potential applicability to many promising areas such as robotics and augmented reality.

3D Object Detection

Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations

no code implementations13 Mar 2020 Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Silvio Savarese, Li Fei-Fei

In the second stage of GTI, we collect a small set of rollouts from the unconditioned stochastic policy of the first stage, and train a goal-directed agent to generalize to novel start and goal configurations.

Imitation Learning

JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset

1 code implementation19 Feb 2020 Abhijeet Shenoi, Mihir Patel, JunYoung Gwak, Patrick Goebel, Amir Sadeghian, Hamid Rezatofighi, Roberto Martín-Martín, Silvio Savarese

In this work we present JRMOT, a novel 3D MOT system that integrates information from RGB images and 3D point clouds to achieve real-time, state-of-the-art tracking performance.

Autonomous Navigation Motion Planning +2

Learning to Navigate Using Mid-Level Visual Priors

1 code implementation23 Dec 2019 Alexander Sax, Jeffrey O. Zhang, Bradley Emi, Amir Zamir, Silvio Savarese, Leonidas Guibas, Jitendra Malik

How much does having visual priors about the world (e. g. the fact that the world is 3D) assist in learning to perform downstream motor tasks (e. g. navigating a complex environment)?

reinforcement-learning Representation Learning

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

no code implementations13 Nov 2019 Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, Dieter Fox

For simple short-horizon manipulation tasks with modest variation in task instances, offline learning from a small set of demonstrations can produce controllers that successfully solve the task.

Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity

no code implementations11 Nov 2019 Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei

We evaluate the quality of our platform, the diversity of demonstrations in our dataset, and the utility of our dataset via quantitative and qualitative analysis.

Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments

1 code implementation30 Oct 2019 Fei Xia, William B. Shen, Chengshu Li, Priya Kasimbeg, Micael Tchapmi, Alexander Toshev, Li Fei-Fei, Roberto Martín-Martín, Silvio Savarese

We present Interactive Gibson Benchmark, the first comprehensive benchmark for training and evaluating Interactive Navigation: robot navigation strategies where physical interaction with objects is allowed and even encouraged to accomplish a task.

Robot Navigation

Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation

no code implementations29 Oct 2019 Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei

The fundamental challenge of planning for multi-step manipulation is to find effective and plausible action sequences that lead to the task goal.

Variational Inference

KETO: Learning Keypoint Representations for Tool Manipulation

no code implementations26 Oct 2019 Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, Silvio Savarese

For this purpose, we present KETO, a framework of learning keypoint representations of tool-based manipulation.


HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators

1 code implementation24 Oct 2019 Chengshu Li, Fei Xia, Roberto Martin-Martin, Silvio Savarese

Different from other HRL solutions, HRL4IN handles the heterogeneous nature of the Interactive Navigation task by creating subgoals in different spaces in different phases of the task.

Hierarchical Reinforcement Learning reinforcement-learning

3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera

1 code implementation ICCV 2019 Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R. Zamir, Martin Fischer, Jitendra Malik, Silvio Savarese

Given a 3D mesh and registered panoramic images, we construct a graph that spans the entire building and includes semantics on objects (e. g., class, material, and other attributes), rooms (e. g., scene category, volume, etc.)

Causal Induction from Visual Observations for Goal Directed Tasks

1 code implementation3 Oct 2019 Suraj Nair, Yuke Zhu, Silvio Savarese, Li Fei-Fei

Causal reasoning has been an indispensable capability for humans and other intelligent animals to interact with the physical world.

Regression Planning Networks

1 code implementation NeurIPS 2019 Danfei Xu, Roberto Martín-Martín, De-An Huang, Yuke Zhu, Silvio Savarese, Li Fei-Fei

Recent learning-to-plan methods have shown promising results on planning directly from observation space.

SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning

no code implementations27 Sep 2019 Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Costa, Silvio Savarese, Li Fei-Fei

We present an overview of SURREAL-System, a reproducible, flexible, and scalable framework for distributed reinforcement learning (RL).

OpenAI Gym reinforcement-learning

AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers

1 code implementation9 Sep 2019 Andrey Kurenkov, Ajay Mandlekar, Roberto Martin-Martin, Silvio Savarese, Animesh Garg

The exploration mechanism used by a Deep Reinforcement Learning (RL) agent plays a key role in determining its sample efficiency.

Situational Fusion of Visual Representation for Visual Navigation

no code implementations ICCV 2019 Bokui Shen, Danfei Xu, Yuke Zhu, Leonidas J. Guibas, Li Fei-Fei, Silvio Savarese

A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities.

Visual Navigation

Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning

no code implementations16 Aug 2019 De-An Huang, Danfei Xu, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei, Juan Carlos Niebles

The key technical challenge is that the symbol grounding is prone to error with limited training data and leads to subsequent symbolic planning failures.

Imitation Learning

Time-Varying Interaction Estimation Using Ensemble Methods

no code implementations25 Jun 2019 Brandon Oselio, Amir Sadeghian, Silvio Savarese, Alfred Hero

Directed information (DI) is a useful tool to explore time-directed interactions in multivariate data.

Ensemble Learning

Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks

no code implementations20 Jun 2019 Roberto Martín-Martín, Michelle A. Lee, Rachel Gardner, Silvio Savarese, Jeannette Bohg, Animesh Garg

This paper studies the effect of different action spaces in deep RL and advocates for Variable Impedance Control in End-effector Space (VICES) as an advantageous action space for constrained and contact-rich tasks.

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

5 code implementations CVPR 2019 Christopher Choy, JunYoung Gwak, Silvio Savarese

To overcome challenges in the 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space.

3D Semantic Segmentation 4D Spatio Temporal Semantic Segmentation +1

Machine Vision for Natural Gas Methane Emissions Detection Using an Infrared Camera

no code implementations1 Apr 2019 Jingfan Wang, Lyne P. Tchapmi, Arvind P. Ravikumara, Mike McGuire, Clay S. Bell, Daniel Zimmerle, Silvio Savarese, Adam R. Brandt

We find that the detection accuracy can reach as high as 99%, the overall detection accuracy can exceed 95% for a case across all leak sizes and imaging distances.

Change Detection Optical Flow Estimation

Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

no code implementations4 Mar 2019 Michael Danielczuk, Andrey Kurenkov, Ashwin Balakrishna, Matthew Matl, David Wang, Roberto Martín-Martín, Animesh Garg, Silvio Savarese, Ken Goldberg

In this paper, we formalize Mechanical Search and study a version where distractor objects are heaped over the target object in a bin.


Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression

10 code implementations CVPR 2019 Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, Silvio Savarese

By incorporating this generalized $IoU$ ($GIoU$) as a loss into the state-of-the art object detection frameworks, we show a consistent improvement on their performance using both the standard, $IoU$ based, and new, $GIoU$ based, performance measures on popular object detection benchmarks such as PASCAL VOC and MS COCO.

Object Detection

Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies

1 code implementation31 Dec 2018 Alexander Sax, Bradley Emi, Amir R. Zamir, Leonidas Guibas, Silvio Savarese, Jitendra Malik

This skill set (hereafter mid-level perception) provides the policy with a more processed state of the world compared to raw images.

Object Detection

Coupled Recurrent Network (CRN)

no code implementations25 Dec 2018 Lin Sun, Kui Jia, Yuejia Shen, Silvio Savarese, Dit Yan Yeung, Bertram E. Shi

To learn from these heterogenous input sources, existing methods reply on two-stream architectural designs that contain independent, parallel streams of Recurrent Neural Networks (RNNs).

Action Recognition Action Recognition In Videos +3

RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation

no code implementations7 Nov 2018 Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, Li Fei-Fei

Imitation Learning has empowered recent advances in learning robotic manipulation tasks by addressing shortcomings of Reinforcement Learning such as exploration and reward specification.

Imitation Learning

Gibson Env: Real-World Perception for Embodied Agents

5 code implementations CVPR 2018 Fei Xia, Amir Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, Silvio Savarese

Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly.

Domain Adaptation General Reinforcement Learning +1

Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration

no code implementations CVPR 2019 De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, Juan Carlos Niebles

We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model.

Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision

no code implementations25 Jun 2018 Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, Silvio Savarese

We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering.

VUNet: Dynamic Scene View Synthesis for Traversability Estimation using an RGB Camera

no code implementations22 Jun 2018 Noriaki Hirose, Amir Sadeghian, Fei Xia, Roberto Martin-Martin, Silvio Savarese

We present VUNet, a novel view(VU) synthesis method for mobile robots in dynamic environments, and its application to the estimation of future traversability.

Autonomous Vehicles

SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

1 code implementation CVPR 2019 Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, S. Hamid Rezatofighi, Silvio Savarese

Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors.

Ranked #4 on Trajectory Prediction on Stanford Drone (ADE (8/12) @K=5 metric)

Self-Driving Cars Trajectory Forecasting

Demo2Vec: Reasoning Object Affordances From Online Videos

no code implementations CVPR 2018 Kuan Fang, Te-Lin Wu, Daniel Yang, Silvio Savarese, Joseph J. Lim

Watching expert demonstrations is an important way for humans and robots to reason about affordances of unseen objects.

Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View

no code implementations CVPR 2018 Shuran Song, Andy Zeng, Angel X. Chang, Manolis Savva, Silvio Savarese, Thomas Funkhouser

We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation ( <=50%) in the form of an RGB-D image.

Generalizing to Unseen Domains via Adversarial Data Augmentation

2 code implementations NeurIPS 2018 Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, Silvio Savarese

Only using training data from a single source distribution, we propose an iterative procedure that augments the dataset with examples from a fictitious target domain that is "hard" under the current model.

Data Augmentation Semantic Segmentation

Deep Learning under Privileged Information Using Heteroscedastic Dropout

1 code implementation CVPR 2018 John Lambert, Ozan Sener, Silvio Savarese

This is what the Learning Under Privileged Information (LUPI) paradigm endeavors to model by utilizing extra knowledge only available during training.

Image Classification Machine Translation +1

Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks

7 code implementations CVPR 2018 Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, Alexandre Alahi

Understanding human motion behavior is critical for autonomous moving platforms (like self-driving cars and social robots) if they are to navigate human-centric environments.

Motion Forecasting Multi-future Trajectory Prediction +2

GONet: A Semi-Supervised Deep Learning Approach For Traversability Estimation

no code implementations8 Mar 2018 Noriaki Hirose, Amir Sadeghian, Marynel Vázquez, Patrick Goebel, Silvio Savarese

We present semi-supervised deep learning approaches for traversability estimation from fisheye images.

Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View

no code implementations12 Dec 2017 Shuran Song, Andy Zeng, Angel X. Chang, Manolis Savva, Silvio Savarese, Thomas Funkhouser

We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation (<= 50%) in the form of an RGB-D image.

CAR-Net: Clairvoyant Attentive Recurrent Network

no code implementations ECCV 2018 Amir Sadeghian, Ferdinand Legros, Maxime Voisin, Ricky Vesel, Alexandre Alahi, Silvio Savarese

We exploit two sources of information: the past motion trajectory of the agent of interest and a wide top-view image of the navigation scene.

Trajectory Forecasting

Adversarial Feature Augmentation for Unsupervised Domain Adaptation

2 code implementations CVPR 2018 Riccardo Volpi, Pietro Morerio, Silvio Savarese, Vittorio Murino

Recent works showed that Generative Adversarial Networks (GANs) can be successfully applied in unsupervised domain adaptation, where, given a labeled source dataset and an unlabeled target dataset, the goal is to train powerful classifiers for the target samples.

Data Augmentation Unsupervised Domain Adaptation

Recurrent Autoregressive Networks for Online Multi-Object Tracking

no code implementations7 Nov 2017 Kuan Fang, Yu Xiang, Xiaocheng Li, Silvio Savarese

The external memory explicitly stores previous inputs of each trajectory in a time window, while the internal memory learns to summarize long-term tracking history and associate detections by processing the external memory.

Frame Multi-Object Tracking +1

Generic 3D Representation via Pose Estimation and Matching

1 code implementation23 Oct 2017 Amir R. Zamir, Tilman Wekel, Pulkit Argrawal, Colin Weil, Jitendra Malik, Silvio Savarese

Though a large body of computer vision research has investigated developing generic semantic representations, efforts towards developing a similar representation for 3D has been limited.

Pose Estimation

SEGCloud: Semantic Segmentation of 3D Point Clouds

no code implementations20 Oct 2017 Lyne P. Tchapmi, Christopher B. Choy, Iro Armeni, JunYoung Gwak, Silvio Savarese

Coarse voxel predictions from a 3D Fully Convolutional NN are transferred back to the raw 3D points via trilinear interpolation.

Neural Task Programming: Learning to Generalize Across Hierarchical Tasks

1 code implementation4 Oct 2017 Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, Silvio Savarese

In this work, we propose a novel robot learning framework called Neural Task Programming (NTP), which bridges the idea of few-shot learning from demonstration and neural program induction.

Few-Shot Learning Program induction

To Go or Not To Go? A Near Unsupervised Learning Approach For Robot Navigation

no code implementations16 Sep 2017 Noriaki Hirose, Amir Sadeghian, Patrick Goebel, Silvio Savarese

It is important for robots to be able to decide whether they can go through a space or not, as they navigate through a dynamic environment.

Anomaly Detection Robot Navigation

Lattice Long Short-Term Memory for Human Action Recognition

no code implementations ICCV 2017 Lin Sun, Kui Jia, Kevin Chen, Dit Yan Yeung, Bertram E. Shi, Silvio Savarese

This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity.

Action Recognition Optical Flow Estimation

DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image

no code implementations11 Aug 2017 Andrey Kurenkov, Jingwei Ji, Animesh Garg, Viraj Mehta, JunYoung Gwak, Christopher Choy, Silvio Savarese

We evaluate our approach on the ShapeNet dataset and show that - (a) the Free-Form Deformation layer is a powerful new building block for Deep Learning models that manipulate 3D data (b) DeformNet uses this FFD layer combined with shape retrieval for smooth and detail-preserving 3D reconstruction of qualitatively plausible point clouds with respect to a single query image (c) compared to other state-of-the-art 3D reconstruction methods, DeformNet quantitatively matches or outperforms their benchmarks by significant margins.

3D Reconstruction 3D Shape Reconstruction

Weakly supervised 3D Reconstruction with Adversarial Constraint

2 code implementations31 May 2017 JunYoung Gwak, Christopher B. Choy, Animesh Garg, Manmohan Chandraker, Silvio Savarese

Supervised 3D reconstruction has witnessed a significant progress through the use of deep neural networks.

3D Reconstruction

Deep View Morphing

no code implementations CVPR 2017 Dinghuang Ji, Junghyun Kwon, Max McFarland, Silvio Savarese

An encoder-decoder network then generates dense correspondences between the rectified images and blending masks to predict the visibility of pixels of the rectified images in the middle view.

Joint 2D-3D-Semantic Data for Indoor Scene Understanding

1 code implementation3 Feb 2017 Iro Armeni, Sasha Sax, Amir R. Zamir, Silvio Savarese

We present a dataset of large-scale indoor spaces that provides a variety of mutually registered modalities from 2D, 2. 5D and 3D domains, with instance-level semantic and geometric annotations.

Scene Understanding

Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies

no code implementations ICCV 2017 Amir Sadeghian, Alexandre Alahi, Silvio Savarese

To address this challenge, we present a structure of Recurrent Neural Networks (RNN) that jointly reasons on multiple cues over a temporal window.

Feedback Networks

1 code implementation CVPR 2017 Amir R. Zamir, Te-Lin Wu, Lin Sun, William Shen, Jitendra Malik, Silvio Savarese

Currently, the most successful learning models in computer vision are based on learning successive representations followed by a decision layer.

Learning Transferrable Representations for Unsupervised Domain Adaptation

no code implementations NeurIPS 2016 Ozan Sener, Hyun Oh Song, Ashutosh Saxena, Silvio Savarese

Supervised learning with large scale labelled datasets and deep layered models has caused a paradigm shift in diverse areas in learning and recognition.

Object Recognition Unsupervised Domain Adaptation

Human Centred Object Co-Segmentation

no code implementations12 Jun 2016 Chenxia Wu, Jiemi Zhang, Ashutosh Saxena, Silvio Savarese

Co-segmentation is the automatic extraction of the common semantic regions given a set of images.

Human-Object Interaction Detection

Universal Correspondence Network

no code implementations NeurIPS 2016 Christopher B. Choy, JunYoung Gwak, Silvio Savarese, Manmohan Chandraker

We present a deep learning framework for accurate visual correspondences and demonstrate its effectiveness for both geometric and semantic matching, spanning across rigid motions to intra-class shape or appearance variations.

Metric Learning Semantic Similarity +1

DeLay: Robust Spatial Layout Estimation for Cluttered Indoor Scenes

no code implementations CVPR 2016 Saumitro Dasgupta, Kuan Fang, Kevin Chen, Silvio Savarese

We consider the problem of estimating the spatial layout of an indoor scene from a monocular RGB image, modeled as the projection of a 3D cuboid.

3D Semantic Parsing of Large-Scale Indoor Spaces

no code implementations CVPR 2016 Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, Silvio Savarese

In this paper, we propose a method for semantic parsing the 3D point cloud of an entire building using a hierarchical approach: first, the raw data is parsed into semantically meaningful spaces (e. g. rooms, etc) that are aligned into a canonical reference coordinate system.

Semantic Parsing

Unsupervised Semantic Action Discovery from Video Collections

no code implementations11 May 2016 Ozan Sener, Amir Roshan Zamir, Chenxia Wu, Silvio Savarese, Ashutosh Saxena

Our method can also provide a textual description for each of the identified semantic steps and video segments.

Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection

1 code implementation16 Apr 2016 Yu Xiang, Wongun Choi, Yuanqing Lin, Silvio Savarese

In CNN-based object detection methods, region proposal becomes a bottleneck when objects exhibit significant scale variation, occlusion or truncation.

2D object detection General Classification +3

Learning to Track at 100 FPS with Deep Regression Networks

3 code implementations6 Apr 2016 David Held, Sebastian Thrun, Silvio Savarese

We propose a method for offline training of neural networks that can track novel objects at test-time at 100 fps.

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

11 code implementations2 Apr 2016 Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, Silvio Savarese

Inspired by the recent success of methods that employ shape priors to achieve robust 3D reconstructions, we propose a novel recurrent neural network architecture that we call the 3D Recurrent Reconstruction Neural Network (3D-R2N2).

3D Object Reconstruction 3D Reconstruction

Knowledge Transfer for Scene-specific Motion Prediction

no code implementations22 Mar 2016 Lamberto Ballan, Francesco Castaldo, Alexandre Alahi, Francesco Palmieri, Silvio Savarese

When given a single frame of the video, humans can not only interpret the content of the scene, but also they are able to forecast the near future.

Frame motion prediction +2

Watch-n-Patch: Unsupervised Learning of Actions and Relations

no code implementations11 Mar 2016 Chenxia Wu, Jiemi Zhang, Ozan Sener, Bart Selman, Silvio Savarese, Ashutosh Saxena

For evaluation, we contribute a new challenging RGB-D activity video dataset recorded by the new Kinect v2, which contains several human daily activities as compositions of multiple actions interacting with different objects.

Action Segmentation

Unsupervised Transductive Domain Adaptation

no code implementations10 Feb 2016 Ozan Sener, Hyun Oh Song, Ashutosh Saxena, Silvio Savarese

We incorporate the domain shift and the transductive target inference into our framework by jointly solving for an asymmetric similarity metric and the optimal transductive target label assignment.

Object Recognition Unsupervised Domain Adaptation

Watch-Bot: Unsupervised Learning for Reminding Humans of Forgotten Actions

no code implementations14 Dec 2015 Chenxia Wu, Jiemi Zhang, Bart Selman, Silvio Savarese, Ashutosh Saxena

We show that our approach not only improves the unsupervised action segmentation and action cluster assignment performance, but also effectively detects the forgotten actions on a challenging human activity RGB-D video dataset.

Action Segmentation

Learning to Track: Online Multi-Object Tracking by Decision Making

no code implementations ICCV 2015 Yu Xiang, Alexandre Alahi, Silvio Savarese

Online Multi-Object Tracking (MOT) has wide applications in time-critical video analysis scenarios, such as robot navigation and autonomous driving.

Autonomous Driving Decision Making +6

Deep Metric Learning via Lifted Structured Feature Embedding

3 code implementations CVPR 2016 Hyun Oh Song, Yu Xiang, Stefanie Jegelka, Silvio Savarese

Additionally, we collected Online Products dataset: 120k images of 23k classes of online products for metric learning.

Metric Learning Structured Prediction

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

2 code implementations CVPR 2016 Ashesh Jain, Amir R. Zamir, Silvio Savarese, Ashutosh Saxena

The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps.

Human Pose Forecasting Skeleton Based Action Recognition

Semantic Cross-View Matching

no code implementations31 Oct 2015 Francesco Castaldo, Amir Zamir, Roland Angst, Francesco Palmieri, Silvio Savarese

In this paper, we therefore explore this idea and propose an automatic method for detecting and representing the semantic information of an RGB image with the goal of performing cross-view matching with a (non-RGB) geographic information system (GIS).

Action Recognition by Hierarchical Mid-level Action Elements

no code implementations ICCV 2015 Tian Lan, Yuke Zhu, Amir Roshan Zamir, Silvio Savarese

Realistic videos of human actions exhibit rich spatiotemporal structures at multiple levels of granularity: an action can always be decomposed into multiple finer-grained elements in both space and time.

Action Parsing Action Recognition

Deep Learning for Single-View Instance Recognition

no code implementations29 Jul 2015 David Held, Sebastian Thrun, Silvio Savarese

We show that feedforward neural networks outperform state-of-the-art methods for recognizing objects from novel viewpoints even when trained from just a single image per object.

Unsupervised Semantic Parsing of Video Collections

no code implementations ICCV 2015 Ozan Sener, Amir Zamir, Silvio Savarese, Ashutosh Saxena

The proposed method is capable of providing a semantic "storyline" of the video composed of its objective steps.

Unsupervised semantic parsing

Data-Driven 3D Voxel Patterns for Object Category Recognition

no code implementations CVPR 2015 Yu Xiang, Wongun Choi, Yuanqing Lin, Silvio Savarese

Despite the great progress achieved in recognizing objects as 2D bounding boxes in images, it is still very challenging to detect occluded objects and estimate the 3D properties of multiple objects from a single image.

Object Recognition Pose Estimation

Watch-n-Patch: Unsupervised Understanding of Actions and Relations

no code implementations CVPR 2015 Chenxia Wu, Jiemi Zhang, Silvio Savarese, Ashutosh Saxena

For evaluation, we also contribute a new challenging RGB-D activity video dataset recorded by the new Kinect v2, which contains several human daily activities as compositions of multiple actions interacted with different objects.

Action Segmentation

Enriching Object Detection With 2D-3D Registration and Continuous Viewpoint Estimation

no code implementations CVPR 2015 Christopher Bongsoo Choy, Michael Stark, Sam Corbett-Davies, Silvio Savarese

We propose an efficient method for synthesizing templates from 3D models that runs on the fly -- that is, it quickly produces detectors for an arbitrary viewpoint of a 3D model without expensive dataset-dependent training or template storage.

Object Detection Pose Estimation +1

A Coarse-to-Fine Model for 3D Pose Estimation and Sub-category Recognition

no code implementations CVPR 2015 Roozbeh Mottaghi, Yu Xiang, Silvio Savarese

Despite the fact that object detection, 3D pose estimation, and sub-category recognition are highly correlated tasks, they are usually addressed independently from each other because of the huge space of parameters.

3D Pose Estimation Object Detection

Learning an Image-based Motion Context for Multiple People Tracking

no code implementations CVPR 2014 Laura Leal-Taixe, Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, Silvio Savarese

We present a novel method for multiple people tracking that leverages a generalized model for capturing interactions among individuals.

Multiple People Tracking

Weakly Supervised Learning of Mid-Level Features with Beta-Bernoulli Process Restricted Boltzmann Machines

no code implementations CVPR 2013 Roni Mittelman, Honglak Lee, Benjamin Kuipers, Silvio Savarese

In order to address this issue, we propose a weakly supervised approach to learn mid-level features, where only class-level supervision is provided during training.

Object Recognition

Dense Object Reconstruction with Semantic Priors

no code implementations CVPR 2013 Sid Yingze Bao, Manmohan Chandraker, Yuanqing Lin, Silvio Savarese

Given multiple images of an unseen instance, we collate information from 2D object detectors to align the structure from motion point cloud with the mean shape, which is subsequently warped and refined to approach the actual shape.

Object Detection Object Reconstruction

Cannot find the paper you are looking for? You can Submit a new open access paper.