Search Results for author:
Found 170 papers, 71 papers with code
Beyond Games: Bringing Exploration to Robots in Real-world
But most importantly, we are able to implement an exploration policy on a robot which learns to interact with objects completely from scratch just using data collected via the differentiable exploration module.
Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips
We tackle the task of reconstructing hand-object interactions from short video clips.
RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking
The grand aim of having a single robot that can manipulate arbitrary objects in diverse settings is at odds with the paucity of robotics datasets.
Evaluating Continual Learning on a Home Robot
Robots in home environments need to be able to learn new skills continuously as data becomes available, becoming ever more capable over time while using as little real-world data as possible.
Train Offline, Test Online: A Real Robot Learning Benchmark
Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data.
Visual Affordance Prediction for Guiding Robot Exploration
Motivated by the intuitive understanding humans have about the space of possible interactions, and the ease with which they can generalize this understanding to previously unseen scenes, we develop an approach for learning visual affordances for guiding robot exploration.
ArK: Augmented Reality with Knowledge Interactive Emergent Ability
In this study, we develop an infinite agent that learns to transfer knowledge memory from general foundation models (e. g. GPT4, DALLE) to novel domains or scenarios for scene understanding and generation in the physical or virtual world.
Affordance Diffusion: Synthesizing Hand-Object Interactions
In contrast, in this work we focus on synthesizing complex interactions (ie, an articulated hand) with a given object.
Zero-Shot Robot Manipulation from Passive Human Videos
Can we learn robot manipulation for everyday tasks, only by watching videos of humans doing arbitrary tasks in different unstructured settings?
Generalized Neural Closure Models with Interpretability
Improving the predictive capability and computational cost of dynamical models is often at the heart of augmenting computational physics with machine learning (ML).
Self-Activating Neural Ensembles for Continual Reinforcement Learning
The ability for an agent to continuously learn new skills without catastrophically forgetting existing knowledge is of critical importance for the development of generally intelligent agents.
Last-Mile Embodied Visual Navigation
Realistic long-horizon tasks like image-goal navigation involve exploratory and exploitative phases.
Bayesian Learning of Coupled Biogeochemical-Physical Models
We develop a Bayesian model learning methodology that allows interpolation in the space of candidate models and discovery of new models from noisy, sparse, and indirect observations, all while estimating state fields and parameter values, as well as the joint PDFs of all learned quantities.
All the Feels: A dexterous hand with large area sensing
High cost and lack of reliability has precluded the widespread adoption of dexterous hands in robotics.
Real World Offline Reinforcement Learning with Realistic Data Source
Offline reinforcement learning (ORL) holds great promise for robot learning due to its ability to learn from arbitrary pre-generated experience.
Learning State-Aware Visual Representations from Audible Interactions
However, learning representations from videos can be challenging.
Ranked #1 on
Long Term Action Anticipation
on Ego4D
(ED@20 Noun metric)
Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Grasps
This paper seeks to escape these constraints, by developing a Pre-Grasp informed Dexterous Manipulation (PGDM) framework that generates diverse dexterous manipulation behaviors, without any task-specific reasoning or hyper-parameter tuning.
Human-to-Robot Imitation in the Wild
We approach the problem of learning by watching humans in the wild.
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?
Task specification is at the core of programming autonomous robots.
What's in your hands? 3D Reconstruction of Generic Objects in Hands
Our work aims to reconstruct hand-held objects given a single RGB image.
Pre-train, Self-train, Distill: A simple recipe for Supersizing 3D Reconstruction
Our final 3D reconstruction model is also capable of zero-shot inference on images from unseen object categories and we empirically show that increasing the number of training categories improves the reconstruction quality.
The Challenges of Continuous Self-Supervised Learning
As a result, SSL holds the promise to learn representations from data in-the-wild, i. e., without the need for finite and static datasets.
R3M: A Universal Visual Representation for Robot Manipulation
We study how visual representations pre-trained on diverse human video data can enable data-efficient learning of downstream robotic manipulation tasks.
The Unsurprising Effectiveness of Pre-Trained Vision Models for Control
In this context, we revisit and study the role of pre-trained visual representations for control, and in particular representations trained on large-scale computer vision datasets.
Interesting Object, Curious Agent: Learning Task-Agnostic Exploration
In this setup, the agent first learns to explore across many environments without any extrinsic goal in a task-agnostic manner.
A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation
Specifying tasks with videos is a powerful technique towards acquiring novel and general robot skills.
ReSkin: versatile, replaceable, lasting tactile skins
Soft sensors have continued growing interest in robotics, due to their ability to enable both passive conformal contact from the material properties and active contact data from the sensor properties.
Dynamic population-based meta-learning for multi-agent communication with natural language
In this work, our goal is to train agents that can coordinate with seen, unseen as well as human partners in a multi-agent communication environment involving natural language.
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
In this work, we present CORA, a platform for Continual Reinforcement Learning Agents that provides benchmarks, baselines, and metrics in a single code package.
No RL, No Simulation: Learning to Navigate without Navigating
Most prior methods for learning navigation policies require access to simulation environments, as they need online policy interaction and rely on ground-truth maps for rewards.
Learning Multi-Objective Curricula for Robotic Policy Learning
Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL).
The Functional Correspondence Problem
For example, what are the correspondences between a bottle and shoe for the task of pounding or the task of pouring.
Wanderlust: Online Continual Object Detection in the Real World
To bridge the gap, we present a new online continual object detection benchmark with an egocentric video dataset, Objects Around Krishna (OAK).
Hierarchical Neural Dynamic Policies
We tackle the problem of generalization to unseen configurations for dynamic tasks in the real world while learning from high-dimensional image input.
Digital-Twin-Based Improvements to Diagnosis, Prognosis, Strategy Assessment, and Discrepancy Checking in a Nearly Autonomous Management and Control System
The Nearly Autonomous Management and Control System (NAMAC) is a comprehensive control system that assists plant operations by furnishing control recommendations to operators in a broad class of situations.
PixelTransformer: Sample Conditioned Signal Generation
We propose a generative model that can infer a distribution for the underlying spatial signal conditioned on sparse samples e. g. plausible images given a few observed pixels.
Learn-to-Race: A Multimodal Control Environment for Autonomous Racing
Existing research on autonomous driving primarily focuses on urban driving, which is insufficient for characterising the complex driving behaviour underlying high-speed racing.
Meta Learning for Multi-agent Communication
Recent works have shown remarkable progress in training artificial agents to understand natural language but are focused on using large amounts of raw data involving huge compute requirements.
Shelf-Supervised Mesh Prediction in the Wild
We first infer a volumetric representation in a canonical frame, along with the camera pose.
droidlet: modular, heterogenous, multi-modal agents
In recent years, there have been significant advances in building end-to-end Machine Learning (ML) systems that learn at scale.
Where2Act: From Pixels to Actions for Articulated 3D Objects
One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment.
Audio-Visual Floorplan Reconstruction
Given only a few glimpses of an environment, how much can we infer about its entire floorplan?
Neural Closure Models for Dynamical Systems
The new "neural closure models" augment low-fidelity models with neural delay differential equations (nDDEs), motivated by the Mori-Zwanzig formulation and the inherent delays in complex dynamical systems.
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image.
Ranked #4 on
Visual Question Answering (VQA)
on A-OKVQA
A 55-line code for large-scale parallel topology optimization in 2D and 3D
This paper presents a 55-line code written in python for 2D and 3D topology optimization (TO) based on the open-source finite element computing software (FEniCS), equipped with various finite element tools and solvers.
Mathematical Software Computational Engineering, Finance, and Science Optimization and Control
Neural Dynamic Policies for End-to-End Sensorimotor Learning
We show that NDPs outperform the prior state-of-the-art in terms of either efficiency or performance across several robotic control tasks for both imitation and reinforcement learning setups.
Ranked #4 on
Meta-Learning
on MT50
Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping
This is largely due to the scale of the datasets both in terms of the number of objects and tasks studied.
Transformers for One-Shot Visual Imitation
Humans are able to seamlessly visually imitate others, by inferring their intentions and using past experience to achieve the same end goal.
Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
We also find that incorporating natural language allows the model to generalize to unseen tasks in a zero-shot setting and to learn quickly from a few demonstrations.
Development and Assessment of a Nearly Autonomous Management and Control System for Advanced Reactors
This paper develops a Nearly Autonomous Management and Control (NAMAC) system for advanced reactors.
Visual Imitation Made Easy
We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector.
Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet.
Implicit Mesh Reconstruction from Unannotated Image Collections
We present an approach to infer the 3D shape, texture, and camera pose for an object from a single RGB image, using only category-level image collections with foreground masks as supervision.
Aligning Videos in Space and Time
During training, given a pair of videos, we compute cycles that connect patches in a given frame in the first video by matching through frames in the second video.
See, Hear, Explore: Curiosity via Audio-Visual Association
Exploration is one of the core challenges in reinforcement learning.
Swoosh! Rattle! Thump! -- Actions that Sound
In this work, we perform the first large-scale study of the interactions between sound and robotic action.
Object Goal Navigation using Goal-Oriented Semantic Exploration
We propose a modular system called, `Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently based on the goal object category.
Ranked #4 on
Robot Navigation
on Habitat 2020 Object Nav test-std
Compositionality and Capacity in Emergent Languages
Our hypothesis is that there should be a specific range of model capacity and channel bandwidth that induces compositional structure in the resulting language and consequently encourages systematic generalization.
Empirically Verifying Hypotheses Using Reinforcement Learning
This paper formulates hypothesis verification as an RL problem.
Learning Robot Skills with Temporal Variational Inference
In this paper, we address the discovery of robotic options from demonstrations in an unsupervised manner.
Semantic Curiosity for Active Visual Learning
Instead, we explore a self-supervised approach for training our exploration policy by introducing a notion of semantic curiosity.
Neural Topological SLAM for Visual Navigation
This paper studies the problem of image-goal navigation which involves navigating to the location indicated by a goal image in a novel previously unseen environment.
Learning to Explore using Active Neural SLAM
The use of learning provides flexibility with respect to input modalities (in the SLAM module), leverages structural regularities of the world (in global policies), and provides robustness to errors in state estimation (in local policies).
Articulation-aware Canonical Surface Mapping
We tackle the tasks of: 1) predicting a Canonical Surface Mapping (CSM) that indicates the mapping from 2D pixels to corresponding points on a canonical template shape, and 2) inferring the articulation and pose of the template corresponding to the input image.
Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects
Our quantitative and qualitative results show that (a) we can predict meaningful forces from videos whose effects lead to accurate imitation of the motions observed, (b) by jointly optimizing for contact point and force prediction, we can improve the performance on both tasks in comparison to independent training, and (c) we can learn a representation from this model that generalizes to novel objects using few shot examples.
Beyond the Camera: Neural Networks in World Coordinates
Eye movement and strategic placement of the visual field onto the retina, gives animals increased resolution of the scene and suppresses distracting information.
Intrinsic Motivation for Encouraging Synergistic Behavior
Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own.
On the interaction between supervision and self-play in emergent communication
A promising approach for teaching artificial agents to use natural language involves using human-in-the-loop training.
Structural Inductive Biases in Emergent Communication
In order to communicate, humans flatten a complex representation of ideas and their attributes into a single word or a sentence.
Towards Graph Representation Learning in Emergent Communication
Recent findings in neuroscience suggest that the human brain represents information in a geometric structure (for instance, through conceptual spaces).
Discovering Motor Programs by Recomposing Demonstrations
In this paper, we present an approach to learn recomposable motor primitives across large-scale and diverse manipulation demonstrations.
ClusterFit: Improving Generalization of Visual Representations
Pre-training convolutional neural networks with weakly-supervised and self-supervised strategies is becoming increasingly popular for several computer vision tasks.
Ranked #52 on
Image Classification
on iNaturalist 2018
Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller
We study a generalized setup for learning from demonstration to build an agent that can manipulate novel objects in unseen scenarios by looking at only a single video of human demonstration from a third-person perspective.
Seeded self-play for language learning
Once the meta-learning agent is able to quickly adapt to each population of agents, it can be deployed in new populations, including populations speaking human language.
Capacity, Bandwidth, and Compositionality in Emergent Language Learning
In this paper, we investigate the learning biases that affect the efficacy and compositionality of emergent languages.
Object-centric Forward Modeling for Model Predictive Control
We present an approach to learn an object-centric forward model, and show that this allows us to plan for sequences of actions to achieve distant desired goals.
Efficient Bimanual Manipulation Using Learned Task Schemas
Our insight is that for many tasks, the learning process can be decomposed into learning a state-independent task schema (a sequence of skills to execute) and a policy to choose the parameterizations of the skills in a state-dependent manner.
Agent as Scientist: Learning to Verify Hypotheses
In order to train the agents, we exploit the underlying structure in the majority of hypotheses -- they can be formulated as triplets (pre-condition, action sequence, post-condition).
Swoosh! Rattle! Thump! - Actions that Sound
In this work, we perform the first large-scale study of the interactions between sound and robotic action.
Dynamics-aware Embeddings
In this paper we consider self-supervised representation learning to improve sample efficiency in reinforcement learning (RL).
Compositional Video Prediction
We present an approach for pixel-level future prediction given an input image of a scene.
Environment Probing Interaction Policies
A key challenge in reinforcement learning (RL) is environment generalization: a policy trained to solve a task in one environment often fails to solve the same task in a slightly different test environment.
Canonical Surface Mapping via Geometric Cycle Consistency
We explore the task of Canonical Surface Mapping (CSM).
PyRobot: An Open-source Robotics Framework for Research and Benchmarking
This paper introduces PyRobot, an open-source robotics framework for research and benchmarking.
Self-Supervised Exploration via Disagreement
In this paper, we propose a formulation for exploration inspired by the work in active learning literature.
3D-RelNet: Joint Object and Relational Network for 3D Prediction
We propose an approach to predict the 3D shape and pose for the objects present in a scene.
Task-Driven Modular Networks for Zero-Shot Compositional Learning
When extending the evaluation to the generalized setting which accounts also for pairs seen during training, we discover that naive baseline methods perform similarly or better than current approaches.
Scaling and Benchmarking Self-Supervised Visual Representation Learning
Self-supervised learning aims to learn representations from the data itself without explicit manual supervision.
Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies
The high-level policy is trained using a sparse, task-dependent reward, and operates by choosing which of the low-level policies to run at any given time.
Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces
To achieve our results, we introduce the Bounce Dataset comprising 5K RGB-D videos of bouncing trajectories of a foam ball to probe surfaces of varying shapes and materials in everyday scenes including homes and offices.
Learning Exploration Policies for Navigation
Numerous past works have tackled the problem of task-driven navigation.
Beyond Grids: Learning Graph Representations for Visual Recognition
Our method further learns to propagate information across all vertices on the graph, and is able to project the learned graph representation back into 2D grids.
Hardware Conditioned Policies for Multi-Robot Transfer Learning
In tasks where knowing the agent dynamics is important for success, we learn an embedding for robot hardware and show that policies conditioned on the encoding of hardware tend to generalize and transfer well.
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
In order to make progress and capture the space of manipulation, we would need to collect a large-scale dataset of diverse tasks such as pouring, opening bottles, stacking objects etc.
Visual Semantic Navigation using Scene Priors
Do we use the semantic/functional priors we have built over years to efficiently search and navigate?
BOLD5000: A public fMRI dataset of 5000 images
Vision science, particularly machine vision, has been revolutionized by introducing large-scale image datasets and statistical learning approaches.
Interpretable Intuitive Physics Model
In order to demonstrate that our system models these underlying physical properties, we train our model on collisions of different shapes (cube, cone, cylinder, spheres etc.)
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
The models trained with our home dataset showed a marked improvement of 43. 7% over a baseline model trained with data collected in lab.
Videos as Space-Time Region Graphs
These nodes are connected by two types of relations: (i) similarity relations capturing the long range dependencies between correlated objects and (ii) spatial-temporal relations capturing the interactions between nearby objects.
Ranked #31 on
Action Classification
on Charades
(using extra training data)
Learning to Grasp Without Seeing
We believe this is the first attempt at learning to grasp with only tactile sensing and without any prior object knowledge.
Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
In this paper we describe the egocentric aspect of the dataset and present annotations for Charades-Ego with 68, 536 activity instances in 68. 8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available.
Actor and Observer: Joint Modeling of First and Third-Person Videos
Several theories in cognitive neuroscience suggest that when people interact with the world, or simulate interactions, they do so from a first-person egocentric perspective, and seamlessly transfer knowledge between third-person (observer) and first-person (actor).
Binge Watching: Scaling Affordance Learning from Sitcoms
In this paper, we tackle the challenge of creating one of the biggest dataset for learning affordances.
Iterative Visual Reasoning Beyond Convolutions
The framework consists of two core modules: a local module that uses spatial memory to store previous beliefs with parallel updates; and a global graph-reasoning module.
Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs
Given a learned knowledge graph (KG), our approach takes as input semantic embeddings for each node (representing visual category).
AI2-THOR: An Interactive 3D Environment for Visual AI
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor. allenai. org.
Learning by Asking Questions
We also show that our model asks questions that generalize to state-of-the-art VQA models and to novel test time distributions.
Sentiment Classification using Images and Label Embeddings
In this project we analysed how much semantic information images carry, and how much value image data can add to sentiment analysis of the text associated with the images.
Visual Features for Context-Aware Speech Recognition
We are working on a corpus of "how-to" videos from the web, and the idea is that an object that can be seen ("car"), or a scene that is being detected ("kitchen") can be used to condition both models on the "context" of the recording, thereby reducing perplexity and improving transcription.
Non-local Neural Networks
Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time.
Ranked #8 on
Action Classification
on Toyota Smarthome dataset
(using extra training data)
Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations
Our contributions are fourfold: (1) To best of our knowledge, we are presenting for the first time a method to learn a 6-DOF grasping net from RGBD input; (2) We build a grasping dataset from demonstrations in virtual reality with rich sensory and interaction annotations.
What Actions are Needed for Understanding Human Actions in Videos?
We present the many kinds of information that will be needed to achieve substantial gains in activity understanding: objects, verbs, intent, and sequential reasoning.
Transitive Invariance for Self-supervised Visual Representation Learning
The objects are connected by two types of edges which correspond to two types of invariance: "different instances but a similar viewpoint and category" and "different viewpoints of the same instance".
CASSL: Curriculum Accelerated Self-Supervised Learning
Recent self-supervised learning approaches focus on using a few thousand data points to learn policies for high-level, low-dimensional action spaces.
Combining Keystroke Dynamics and Face Recognition for User Verification
Keystroke Dynamics is a novel Biometric Technique; it is not only unobtrusive, but also transparent and inexpensive.
Temporal Dynamic Graph LSTM for Action-driven Video Object Detection
A common issue, however, is that objects of interest that are not involved in human actions are often absent in global action descriptions known as "missing label".
Ranked #3 on
Weakly Supervised Object Detection
on Charades
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
What will happen if we increase the dataset size by 10x or 100x?
Ranked #2 on
Semantic Segmentation
on PASCAL VOC 2007
From Red Wine to Red Tomato: Composition With Context
In this paper, we present a simple method that respects contextuality in order to compose classifiers of known visual concepts.
Visual Semantic Planning using Deep Successor Representations
A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world.
WebVision Challenge: Visual Learning and Understanding With Web Data
The 2017 WebVision challenge consists of two tracks, the image classification task on WebVision test set, and the transfer learning task on PASCAL VOC 2012 dataset.
The Pose Knows: Video Forecasting by Generating Pose Futures
First we explicitly model the high level structure of active objects in the scene---humans---and use a VAE to model the possible future movements of humans in the pose space.
Ranked #2 on
Human Pose Forecasting
on Human3.6M
(CMD metric)
Spatial Memory for Context Reasoning in Object Detection
On the other hand, modeling object-object relationships requires {\bf spatial} reasoning -- not only do we need a memory to store the spatial layout, but also a effective reasoning module to extract spatial patterns.
What's in a Question: Using Visual Questions as a Form of Supervision
For instance, the question "what is the breed of the dog?"
A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection
We propose to learn an adversarial network that generates examples with occlusions and deformations.
Ranked #19 on
Object Detection
on PASCAL VOC 2007
(using extra training data)
ActionVLAD: Learning spatio-temporal aggregation for action classification
In this work, we introduce a new video representation for action classification that aggregates local convolutional features across the entire spatio-temporal extent of the video.
Robust Adversarial Reinforcement Learning
Deep neural networks coupled with fast simulation and improved computation have led to recent successes in the field of reinforcement learning (RL).
PixelNet: Representation of the pixels, by the pixels, and for the pixels
We explore design principles for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation.
An Implementation of Faster RCNN with Study for Region Sampling
We adapted the join-training scheme of Faster RCNN framework from Caffe to TensorFlow as a baseline implementation for object detection.
Learning From Noisy Large-Scale Datasets With Minimal Supervision
For the small clean set of annotations we use a quarter of the validation set with ~40k images.
From Images to 3D Shape Attributes
Our first objective is to infer these 3D shape attributes from a single image.
Beyond Skip Connections: Top-Down Modulation for Object Detection
But most of these fine details are lost in the early convolutional layers.
Ranked #218 on
Object Detection
on COCO test-dev
Asynchronous Temporal Fields for Action Recognition
Actions are more than just movements and trajectories: we cook to eat and we hold a cup to drink from it.
Ranked #15 on
Action Detection
on Charades
The More You Know: Using Knowledge Graphs for Image Classification
One characteristic that sets humans apart from modern learning-based computer vision algorithms is the ability to acquire knowledge about the world and use that knowledge to reason about the visual world.
Supervision via Competition: Robot Adversaries for Learning Tasks
Due to large number of experiences required for training, most of these approaches use a self-supervised paradigm: using sensors to measure success/failure.
Learning to Push by Grasping: Using multiple tasks for effective learning
The argument of the difficulty in scalability to multiple tasks is well founded, since training these tasks often require hundreds or thousands of examples.
PixelNet: Towards a General Pixel-level Architecture
We explore architectures for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation.
Pose from Action: Unsupervised Learning of Pose Features based on Motion
We propose an unsupervised method to learn pose features from videos that exploits a signal which is complementary to appearance and can be used as supervision: motion.
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine.
Much Ado About Time: Exhaustive Annotation of Temporal Data
We conclude that the optimal strategy is to ask as many questions as possible in a HIT (up to 52 binary questions after watching a 30-second video clip in our experiments).
An Uncertain Future: Forecasting from Static Images using Variational Autoencoders
We show that our method is able to successfully predict events in a wide variety of scenes and can produce multiple different predictions when the future is ambiguous.
3D Shape Attributes
In this paper we investigate 3D attributes as a means to understand the shape of an object in a single image.
Learning Visual Storylines with Skipping Recurrent Neural Networks
What does a typical visit to Paris look like?
Training Region-based Object Detectors with Online Hard Example Mining
Our motivation is the same as it has always been -- detection datasets contain an overwhelming number of easy examples and a small number of hard examples.
Ranked #6 on
Face Verification
on Trillion Pairs Dataset
Cross-stitch Networks for Multi-task Learning
In this paper, we propose a principled approach to learn shared representations in ConvNets using multi-task learning.
Ranked #97 on
Semantic Segmentation
on NYU Depth v2
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
Each video is annotated by multiple free-text descriptions, action labels, action intervals and classes of interacted objects.
The Curious Robot: Learning Visual Representations via Physical Interactions
We argue that biological agents use physical interactions with the world to learn visual representations unlike current vision systems which just use passive observations (images and videos downloaded from web).
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction
We introduce an approach that leverages surface normal predictions, along with appearance cues, to retrieve 3D models for objects depicted in 2D still images from a large CAD object library.
Learning a Predictable and Generative Vector Representation for Objects
The network consists of two components: (a) an autoencoder that ensures the representation is generative; and (b) a convolutional network that ensures the representation is predictable.
"What happens if..." Learning to Predict the Effect of Forces in Images
To build a dataset of forces in scenes, we reconstructed all images in SUN RGB-D dataset in a physics simulator to estimate the physical movements of objects caused by external forces applied to them.
Generative Image Modeling using Style and Structure Adversarial Networks
Current generative frameworks use end-to-end learning and generate images by sampling from uniform noise distribution.
Actions ~ Transformations
In this paper, we propose a novel representation for actions by modeling an action as a transformation which changes the state of the environment before the action happens (precondition) to the state after the action (effect).
Single Image 3D Without a Single 3D Image
Do we really need 3D labels in order to learn how to predict 3D?
Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours
Our experiments clearly show the benefit of using large-scale datasets (and multi-stage training) for the task of grasping.
Sense Discovery via Co-Clustering on Images and Text
We present a co-clustering framework that can be used to discover multiple semantic and visual senses of a given Noun Phrase (NP).
Unsupervised Visual Representation Learning by Context Prediction
This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation.
Webly Supervised Learning of Convolutional Networks
Specifically inspired by curriculum learning, we present a two-step approach for CNN training.
In Defense of the Direct Perception of Affordances
The field of functional recognition or affordance estimation from images has seen a revival in recent years.
Unsupervised Learning of Visual Representations using Videos
Is strong supervision necessary for learning a good visual representation?
Dense Optical Flow Prediction from a Static Image
Because our CNN model makes no assumptions about the underlying scene, it can predict future optical flow on a diverse set of scenarios.
Mid-level Elements for Object Detection
Building on the success of recent discriminative mid-level elements, we propose a surprisingly simple approach for object detection which performs comparable to the current state-of-the-art approaches on PASCAL VOC comp-3 detection challenge (no external data).
Transferring Rich Feature Hierarchies for Robust Visual Tracking
To fit the characteristics of object tracking, we first pre-train the CNN to recognize what is an object, and then propose to generate a probability map instead of producing a simple class label.
Designing Deep Networks for Surface Normal Estimation
We show by incorporating several constraints (man-made, manhattan world) and meaningful intermediate representations (room layout, edge labels) in the architecture leads to state of the art performance on surface normal estimation.
Enriching Visual Knowledge Bases via Object Discovery and Segmentation
In this paper, we propose to enrich these knowledge bases by automatically discovering objects and their segmentations from noisy Internet images.
Patch to the Future: Unsupervised Visual Prediction
In this paper we present a conceptually simple but surprisingly powerful method for visual prediction which combines the effectiveness of mid-level visual elements with temporal modeling.
Mid-level Visual Element Discovery as Discriminative Mode Seeking
We also propose the Purity-Coverage plot as a principled way of experimentally analyzing and evaluating different visual discovery approaches, and compare our method against prior work on the Paris Street View dataset.
Representing Videos Using Mid-level Discriminative Patches
representation for videos based on mid-level discriminative spatio-temporal patches.
Estimating Spatial Layout of Rooms using Volumetric Reasoning about Objects and Surfaces
There has been a recent push in extraction of 3D spatial layout of scenes.
A ``Shape Aware'' Model for semi-supervised Learning of Objects and its Context
Using an analogous reasoning, we present an approach that combines bag-of-words and spatial models to perform semantic and syntactic analysis for recognition of an object based on its internal appearance and its context.
