Search Results for author:
Found 217 papers, 104 papers with code
Embodied AI Agents: Modeling the World
This paper describes our research on AI agents embodied in visual, virtual or physical forms, enabling them to interact with both users and their environments.
Whole-Body Conditioned Egocentric Video Prediction
We train models to Predict Ego-centric Video from human Actions (PEVA), given the past video and an action represented by the relative 3D body pose.
SkillBlender: Towards Versatile Humanoid Whole-Body Loco-Manipulation via Skill Blending
To that end, we introduce SkillBlender, a novel hierarchical reinforcement learning framework for versatile humanoid loco-manipulation.
Rodrigues Network for Learning Robot Actions
Understanding and predicting articulated actions is important in robot learning.
AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time
$\alpha$1 first introduces $\alpha$ moment, which represents the scaled thinking phase with a universal parameter $\alpha$.
DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy
To improve generalization across diverse garment shapes and deformations, we propose a Hierarchical gArment-manipuLation pOlicy (HALO).
Visual Imitation Enables Contextual Humanoid Control
How can we teach humanoids to climb staircases and sit on chairs using the surrounding environment context?
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction
Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions.
Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids
This work investigates the key challenges in applying reinforcement learning to solve a collection of contact-rich manipulation tasks on a humanoid embodiment.
Poly-Autoregressive Prediction for Modeling Interactions
At its core, PAR represents the behavior of all agents as a sequence of tokens, each representing an agent's state at a specific timestep.
DexterityGen: Foundation Controller for Unprecedented Dexterity
In the real world, we use human teleoperation as a prompt to the controller to produce highly dexterous behavior.
From Simple to Complex Skills: The Case of In-Hand Object Reorientation
This hierarchical policy learns to select which low-level skill to execute based on feedback from both the environment and the low-level skill policies themselves.
An Empirical Study of Autoregressive Pre-training from Videos
Our models are pre-trained on a diverse dataset of videos and images comprising over 1 trillion visual tokens.
Gaussian Masked Autoencoders
Our approach, named Gaussian Masked Autoencoder, or GMAE, aims to learn semantic abstractions and spatial understanding jointly.
Reconstructing People, Places, and Cameras
We present "Humans and Structure from Motion" (HSfM), a method for jointly reconstructing multiple human meshes, scene point clouds, and camera parameters in a metric world coordinate system from a sparse set of uncalibrated multi-view images featuring people.
Learning from Massive Human Videos for Universal Humanoid Pose Control
Scalable learning of humanoid robots is crucial for their deployment in real-world applications.
Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignment
While reinforcement learning from human feedback (RLHF) has become the predominant mechanism for alignment in non-embodied domains like large language models, it has not seen the same success in aligning visuomotor policies due to the prohibitive amount of human feedback required to learn visual reward functions.
Scaling Properties of Diffusion Models for Perceptual Tasks
In this paper, we argue that iterative computation with diffusion models offers a powerful paradigm for not only generation but also visual perception tasks.
Digitizing Touch with an Artificial Multimodal Fingertip
These results demonstrate the possibility of digitizing touch with superhuman performance.
Learning Humanoid Locomotion over Challenging Terrain
Humanoid robots can, in principle, use their legs to go almost anywhere.
Estimating Body and Hand Motion in an Ego-sensed World
We present EgoAllo, a system for human motion estimation from a head-mounted device.
Hand-Object Interaction Pretraining from Videos
We present an approach to learn general robot manipulation priors from 3D hand-object interaction trajectories.
Synergy and Synchrony in Couple Dances
Our contributions are a demonstration of the advantages of socially conditioned future motion prediction and an in-the-wild, couple dance video dataset to enable future research in this direction.
Wolf: Captioning Everything with a World Summarization Framework
Finally, we establish a benchmark for video captioning and introduce a leaderboard, aiming to accelerate advancements in video understanding, captioning, and data alignment.
Lessons from Learning to Spin "Pens"
This serves two purposes: 1) pre-training a sensorimotor policy in simulation; 2) conducting open-loop trajectory replay in the real world.
Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensing
We introduce a sensor model for tactile skin that enables zero-shot sim-to-real transfer of ternary shear and binary normal forces.
Learning Visuotactile Skills with Two Multifingered Hands
Two significant challenges exist: the lack of an affordable and accessible teleoperation system suitable for a dual-arm setup with multifingered hands, and the scarcity of multifingered hand hardware equipped with touch sensing.
Reconstructing Hand-Held Objects in 3D from Images and Videos
In order to obtain the best performing single frame model, we first present MCC-Hand-Object (MCC-HO), which jointly reconstructs hand and object geometry given a single RGB image and inferred 3D hand as inputs.
AutoEval Done Right: Using Synthetic Data for Model Evaluation
The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming.
xT: Nested Tokenization for Larger Context in Large Images
There are many downstream applications in which global context matters as much as high frequency details, such as in real-world satellite imagery; in such cases researchers have to make the uncomfortable choice of which information to discard.
Twisting Lids Off with Two Hands
Manipulating objects with two multi-fingered hands has been a long-standing challenge in robotics, due to the contact-rich nature of many manipulation tasks and the complexity inherent in coordinating a high-dimensional bimanual system.
Humanoid Locomotion as Next Token Prediction
We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language.
Synthesizing Moving People with 3D Control
This disentangled approach allows our method to generate a sequence of images that are faithful to the target motion in the 3D pose and, to the input image in terms of visual similarity.
Dr$^2$Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning
We use two coefficients on either type of residual connections respectively, and introduce a dynamic training strategy that seamlessly transitions the pretrained model to a reversible network with much higher numerical precision.
Dr2Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning
We use two coefficients on either type of residual connections respectively and introduce a dynamic training strategy that seamlessly transitions the pretrained model to a reversible network with much higher numerical precision.
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation
Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity.
Adaptive Human Trajectory Prediction via Latent Corridors
We formalize the problem of scene-specific adaptive trajectory prediction and propose a new adaptation approach inspired by prompt tuning called latent corridors.
Reconstructing Hands in 3D with Transformers
The key to HaMeR's success lies in scaling up both the data used for training and the capacity of the deep network for hand reconstruction.
Sequential Modeling Enables Scalable Learning for Large Vision Models
We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts
The key idea is the design of switching policies that can take conformal quantiles as input, which we define as conformal policy learning, that allows robots to detect distribution shifts with formal statistical guarantees.
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
We present Habitat 3. 0: a simulation platform for studying collaborative human-robot tasks in home environments.
Interactive Task Planning with Language Models
To tackle this, we propose a simple framework that achieves interactive task planning with language models by incorporating both high-level planning and low-level skill execution through function calling, leveraging pretrained vision models to ground the scene in language.
What Matters to You? Towards Visual Representation Alignment for Robot Learning
When operating in service of people, robots need to optimize rewards aligned with end-user preferences.
Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions
We introduce Conformal Decision Theory, a framework for producing safe autonomous decisions despite imperfect machine learning predictions.
General In-Hand Object Rotation with Vision and Touch
We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs.
Learning Vision-based Pursuit-Evasion Robot Policies
We find that the quality of the supervision signal for the partially-observable pursuer policy depends on two key factors: the balance of diversity and optimality of the evader's behavior and the strength of the modeling assumptions in the fully-observable policy.
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding
We introduce EgoSchema, a very long-form video question-answering dataset, and benchmark to evaluate long video understanding capabilities of modern vision and language systems.
Robot Learning with Sensorimotor Pre-training
We present a self-supervised sensorimotor pre-training approach for robotics.
Learning Space-Time Semantic Correspondences
We propose a new task of space-time semantic correspondence prediction in videos.
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance.
Ranked #1 on
Image Classification
on iNaturalist 2019
(using extra training data)
Humans in 4D: Reconstructing and Tracking Humans with Transformers
To analyze video, we use 3D reconstructions from HMR 2. 0 as input to a tracking system that operates in 3D.
Ranked #3 on
Pose Tracking
on PoseTrack2018
On the Benefits of 3D Pose and Tracking for Human Action Recognition
Subsequently, we propose a Lagrangian Action Recognition model by fusing 3D pose and contextualized appearance over tracklets.
Ranked #1 on
Action Recognition
on AVA v2.2
(using extra training data)
Navigating to Objects Specified by Images
Our modular method solves sub-tasks of exploration, goal instance re-identification, goal localization, and local navigation.
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average).
Real-World Humanoid Locomotion with Reinforcement Learning
Humanoid robots that can autonomously operate in diverse environments have the potential to help address labour shortages in factories, assist elderly at homes, and colonize new planets.
Decoupling Human and Camera Motion from Videos in the Wild
Our method robustly recovers the global 3D trajectories of people in challenging in-the-wild videos, such as PoseTrack.
Reversible Vision Transformers
Reversible Vision Transformers achieve a reduced memory footprint of up to 15. 5x at roughly identical model complexity, parameters and accuracy, demonstrating the promise of reversible vision transformers as an efficient backbone for hardware resource limited training regimes.
Multiview Compressive Coding for 3D Reconstruction
We introduce a simple framework that operates on 3D points of single objects or whole scenes coupled with category-agnostic large-scale training from diverse RGB-D videos.
CA$^2$T-Net: Category-Agnostic 3D Articulation Transfer from Single Image
We present a neural network approach to transfer the motion from a single image of an articulated object to a rest-state (i. e., unarticulated) 3D model.
Re-evaluating the Need for Multimodal Signals in Unsupervised Grammar Induction
Are multimodal inputs necessary for grammar induction?
MAViL: Masked Audio-Video Learners
We present Masked Audio-Video Learners (MAViL) to train audio-visual representations.
Navigating to Objects in the Real World
In contrast, end-to-end learning does not, dropping from 77% simulation to 23% real-world success rate due to a large image domain gap between simulation and reality.
Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances
We consider the problem of embodied visual navigation given an image-goal (ImageNav) where an agent is initialized in an unfamiliar environment and tasked with navigating to a location 'described' by an image.
Learning to Imitate Object Interactions from Internet Videos
In this paper we make two main contributions: (1) a novel reconstruction technique RHOV (Reconstructing Hands and Objects from Videos), which reconstructs 4D trajectories of both the hand and the object using 2D image cues and temporal smoothness constraints; (2) a system for imitating object interactions in a physics simulator with reinforcement learning.
Legged Locomotion in Challenging Terrains using Egocentric Vision
Animals are capable of precise and agile locomotion using vision.
Learning Visual Locomotion with Cross-Modal Supervision
In this work, we show how to learn a visual walking policy that only uses a monocular RGB camera and proprioception.
In-Hand Object Rotation via Rapid Motor Adaptation
Generalized in-hand manipulation has long been an unsolved challenge of robotics.
Real-World Robot Learning with Masked Visual Pre-training
Finally, we train a 307M parameter vision transformer on a massive collection of 4. 5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.
Learning to Learn with Generative Models of Neural Network Checkpoints
We explore a data-driven approach for learning to optimize neural networks.
Learning a Single Near-hover Position Controller for Vastly Different Quadcopters
This paper proposes an adaptive near-hover position controller for quadcopters, which can be deployed to quadcopters of very different mass, size and motor constants, and also shows rapid adaptation to unknown disturbances during runtime.
Multi-skill Mobile Manipulation for Object Rearrangement
To tackle the entire task, prior work chains multiple stationary manipulation skills with a point-goal navigation skill, which are learned individually on subtasks.
Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
After re-examining the design choices for both the macro and micro-architecture of Conformer, we propose Squeezeformer which consistently outperforms the state-of-the-art ASR models under the same training schemes.
Ranked #38 on
Speech Recognition
on LibriSpeech test-clean
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
Adapting Rapid Motor Adaptation for Bipedal Robots
In this work, we leverage recent advances in rapid adaptation for locomotion control, and extend them to work on bipedal robots.
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity
From PA we construct a large set of pseudo-ground-truth instance masks; combined with human-annotated instance masks we train GGNs and significantly outperform the SOTA on open-world instance segmentation on various benchmarks including COCO, LVIS, ADE20K, and UVO.
Masked Visual Pre-training for Motor Control
This paper shows that self-supervised visual pre-training from real-world images is effective for learning motor control tasks from pixels.
Ranked #1 on
Robot Manipulation Generalization
on The COLOSSEUM
Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging
Image-to-image regression is an important learning task, used frequently in biological imaging.
PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning
We propose Potential functions for ObjectGoal Navigation with Interaction-free learning (PONI), a modular approach that disentangles the skills of `where to look?'
MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition
Instead of trying to process more frames at once like most existing methods, we propose to process videos in an online fashion and cache "memory" at each iteration.
Ranked #6 on
Action Anticipation
on EPIC-KITCHENS-100
(using extra training data)
Tracking People by Predicting 3D Appearance, Location and Pose
For a future frame, we compute the similarity between the predicted state of a tracklet and the single frame observations in a probabilistic manner.
Tracking People by Predicting 3D Appearance, Location & Pose
For a future frame, we compute the similarity between the predicted state of a tracklet and the single frame observations in a probabilistic manner.
Coupling Vision and Proprioception for Navigation of Legged Robots
A safety advisor module adds sensed unexpected obstacles to the occupancy map and environment-determined speed limits to the velocity command generator.
SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency
The observations gathered by this exploration policy are labelled using 3D consistency and used to improve the perception model.
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection
In this paper, we study Multiscale Vision Transformers (MViTv2) as a unified architecture for image and video classification, as well as object detection.
Ranked #1 on
Action Classification
on Kinetics-600
(GFLOPs metric)
Differentiable Spatial Planning using Transformers
We consider the problem of spatial path planning.
PyTorchVideo: A Deep Learning Library for Video Understanding
We introduce PyTorchVideo, an open-source deep-learning library that provides a rich set of modular, efficient, and reproducible components for a variety of video understanding tasks, including classification, detection, self-supervised learning, and low-level processing.
Tracking People with 3D Representations
We find that 3D representations are more effective than 2D representations for tracking in these settings, and we obtain state-of-the-art performance.
Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots
We demonstrate that learning to minimize energy consumption plays a key role in the emergence of natural locomotion gaits at different speeds in real quadruped robots.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding
ABO contains product catalog images, metadata, and artist-created 3D models with complex geometries and physically-based materials that correspond to real, household objects.
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans
This paper introduces a pipeline to parametrically sample and render multi-task vision datasets from comprehensive 3D scans from the real world.
Differentiable Stereopsis: Meshes from multiple views using differentiable rendering
We propose Differentiable Stereopsis, a multi-view stereo approach that reconstructs shape and texture from few input views and noisy cameras.
Active 3D Shape Reconstruction from Vision and Touch
In this paper, we focus on this problem and introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2)a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile signals; and 3) a set of data-driven solutions with either tactile or visuotactile priors to guide the shape exploration.
RMA: Rapid Motor Adaptation for Legged Robots
Successful real-world deployment of legged robots would require them to adapt in real-time to unseen scenarios like changing terrains, changing payloads, wear and tear.
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
We introduce Habitat 2. 0 (H2. 0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios.
Multiscale Vision Transformers
We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10x more costly in computation and parameters.
Ranked #14 on
Action Classification
on Charades
Distribution-Free, Risk-Controlling Prediction Sets
While improving prediction accuracy has been the focus of machine learning in recent years, this alone does not suffice for reliable decision-making.
Human Mesh Recovery from Multiple Shots
The tools we develop open the door to processing and analyzing in 3D content from a large library of edited media, which could be helpful for many downstream applications.
Reconstructing Hand-Object Interactions in the Wild
In this work we explore reconstructing hand-object interactions in the wild.
From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting
Uncertainty in future trajectories stems from two sources: (a) sources that are known to the agent but unknown to the model, such as long term goals and (b)sources that are unknown to both the agent & the model, such as intent of other agents & irreducible randomness indecisions.
Ranked #2 on
Trajectory Prediction
on ETH/UCY
Better Knowledge Retention through Metric Learning
In continual learning, new categories may be introduced over time, and an ideal learning system should perform well on both the original categories and the new categories.
Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
Vision-based robotics often separates the control loop into one module for perception and a separate module for control.
Rearrangement: A Challenge for Embodied AI
In the rearrangement task, the goal is to bring a given physical environment into a specified state.
Shape, Illumination, and Reflectance from Shading
A fundamental problem in computer vision is that of inferring the intrinsic, 3D structure of the world from flat, 2D images of that world.
Uncertainty Sets for Image Classifiers using Conformal Prediction
Convolutional image classifiers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings.
Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
Learning long-term dynamics models is the key to understanding physical common sense.
Ranked #1 on
Visual Reasoning
on PHYRE-1B-Within
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild
We present a method that infers spatial arrangements and shapes of humans and objects in a globally consistent 3D scene, all from a single image in-the-wild captured in an uncontrolled environment.
Ranked #3 on
3D Object Reconstruction
on BEHAVE
Shape and Viewpoint without Keypoints
We present a learning framework that learns to recover the 3D shape, pose and texture from a single image, trained on an image collection without any ground truth 3D shape, multi-view, camera viewpoints or keypoint supervision.
Long-term Human Motion Prediction with Scene Context
Human movement is goal-directed and influenced by the spatial layout of the objects in the scene.
3D Shape Reconstruction from Vision and Touch
When a toddler is presented a new toy, their instinctual behaviour is to pick it upand inspect it with their hand and eyes in tandem, clearly searching over its surface to properly understand what they are playing with.
Deep Isometric Learning for Visual Recognition
Initialization, normalization, and skip connections are believed to be three indispensable techniques for training very deep convolutional neural networks and obtaining state-of-the-art performance.
Robust Learning Through Cross-Task Consistency
Visual perception entails solving a wide set of tasks, e. g., object detection, depth estimation, etc.
Multimodal Image Synthesis with Conditional Implicit Maximum Likelihood Estimation
Many tasks in computer vision and graphics fall within the framework of conditional image synthesis.
Inclusive GAN: Improving Data and Minority Coverage in Generative Models
Generative Adversarial Networks (GANs) have brought about rapid progress towards generating photorealistic images.
State-Only Imitation Learning for Dexterous Manipulation
To tackle this setting, we train an inverse dynamics model and use it to predict actions for state-only demonstrations.
It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction
In this work, we present Predicted Endpoint Conditioned Network (PECNet) for flexible human trajectory prediction.
Ranked #1 on
Multi Future Trajectory Prediction
on ETH/UCY
Audiovisual SlowFast Networks for Video Recognition
We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception.
Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks
When training a neural network for a desired task, one may prefer to adapt a pre-trained network rather than starting from randomly initialized weights.
Learning to Navigate Using Mid-Level Visual Priors
How much does having visual priors about the world (e. g. the fact that the world is 3D) assist in learning to perform downstream motor tasks (e. g. navigating a complex environment)?
Approximate Feature Collisions in Neural Nets
Work on adversarial examples has shown that neural nets are surprisingly sensitive to adversarially chosen changes of small magnitude.
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera
Given a 3D mesh and registered panoramic images, we construct a graph that spans the entire building and includes semantics on objects (e. g., class, material, and other attributes), rooms (e. g., scene category, volume, etc.)
Predicting 3D Human Dynamics from Video
In this work, we present perhaps the first approach for predicting a future 3D mesh model sequence of a person from past video input.
Learning Individual Styles of Conversational Gesture
Specifically, we perform cross-modal translation from "in-the-wild'' monologue speech of a single speaker to their hand and arm motion.
Ranked #4 on
Gesture Generation
on BEAT
Mesh R-CNN
We propose a system that detects objects in real-world images and produces a triangle mesh giving the full 3D shape of each detected object.
Ranked #1 on
3D Shape Modeling
on Pix3D S2
Learning Navigation Subroutines from Egocentric Videos
We demonstrate our proposed approach in context of navigation, and show that we can successfully learn consistent and diverse visuomotor subroutines from passive egocentric videos.
Which Tasks Should Be Learned Together in Multi-task Learning?
Many computer vision applications require solving multiple tasks in real-time.
Accelerated Sparse Recovery Under Structured Measurements
Extensive work on compressed sensing has yielded a rich collection of sparse recovery algorithms, each making different tradeoffs between recovery condition and computational efficiency.
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors
However, it is difficult and costly to segment objects in novel categories because a large number of mask annotations is required.
Habitat: A Platform for Embodied AI Research
We present Habitat, a platform for research in embodied artificial intelligence (AI).
Ranked #2 on
PointGoal Navigation
on Gibson PointGoal Navigation
Combining Optimal Control and Learning for Visual Navigation in Novel Environments
Model-based control is a popular paradigm for robot navigation because it can leverage a known dynamics model to efficiently plan robust robot trajectories.
Trajectory Normalized Gradients for Distributed Optimization
Recently, researchers proposed various low-precision gradient compression, for efficient communication in large-scale distributed optimization.
Learning Independent Object Motion from Unlabelled Stereoscopic Videos
Unlike prior learning based work which has focused on predicting dense pixel-wise optical flow field and/or a depth map for each image, we propose to predict object instance specific 3D scene flow maps and instance masks from which we are able to derive the motion direction and speed for each object instance.
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
This skill set (hereafter mid-level perception) provides the policy with a more processed state of the world compared to raw images.
Non-Adversarial Image Synthesis with Generative Latent Nearest Neighbors
GLANN combines the strengths of IMLE and GLO in a way that overcomes the main drawbacks of each method.
SlowFast Networks for Video Recognition
We present SlowFast networks for video recognition.
Ranked #4 on
Action Recognition
on AVA v2.1
Learning 3D Human Dynamics from Video
We present a framework that can similarly learn a representation of 3D dynamics of humans from video via a simple but effective temporal encoding of image features.
Ranked #17 on
3D Human Pose Estimation
on 3DPW
(Acceleration Error metric)
Visual Memory for Robust Path Following
Equipped with this abstraction, a second network observes the world and decides how to act to retrace the path under noisy actuation and a changing environment.
Are All Training Examples Created Equal? An Empirical Study
Modern computer vision algorithms often rely on very large training datasets.
On the Implicit Assumptions of GANs
Generative adversarial nets (GANs) have generated a lot of excitement.
Diverse Image Synthesis from Semantic Layouts via Conditional IMLE
Most existing methods for conditional image synthesis are only able to generate a single plausible image for any given input, or at best a fixed number of plausible images.
SFV: Reinforcement Learning of Physical Skills from Videos
In this paper, we propose a method that enables physically simulated characters to learn skills from videos (SFV).
Super-Resolution via Conditional Implicit Maximum Likelihood Estimation
Single-image super-resolution (SISR) is a canonical problem with diverse applications.
A Study of Robustness of Neural Nets Using Approximate Feature Collisions
In recent years, various studies have focused on the robustness of neural nets.
Implicit Maximum Likelihood Estimation
Implicit probabilistic models are models defined naturally in terms of a sampling procedure and often induces a likelihood function that cannot be expressed explicitly.
Cost-Sensitive Active Learning for Intracranial Hemorrhage Detection
Deep learning for clinical applications is subject to stringent performance requirements, which raises a need for large labeled datasets.
Gibson Env: Real-World Perception for Embodied Agents
Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly.
On Evaluation of Embodied Navigation Agents
Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence.
Learning Instance Segmentation by Interaction
The agent uses its current segmentation model to infer pixels that constitute objects and refines the segmentation model by interacting with these pixels.
PatchFCN for Intracranial Hemorrhage Detection
This paper studies the problem of detecting and segmenting acute intracranial hemorrhage on head computed tomography (CT) scans.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
This model -- a deep, multimodal convolutional network -- predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions.
Taskonomy: Disentangling Task Transfer Learning
The product is a computational taxonomic map for task transfer learning.
Zero-Shot Visual Imitation
In our framework, the role of the expert is only to communicate the goals (i. e., what to imitate) during inference.
Learning Category-Specific Mesh Reconstruction from Image Collections
The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation.
Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction
We present a framework for learning single-view shape and pose prediction without using direct supervision for either.
Unifying Map and Landmark Based Representations for Visual Navigation
This works presents a formulation for visual navigation that unifies map based spatial reasoning and path planning, with landmark based robust plan execution in noisy environments.
End-to-end Recovery of Human Shape and Pose
The main objective is to minimize the reprojection loss of keypoints, which allow our model to be trained using images in-the-wild that only have ground truth 2D annotations.
Ranked #1 on
Weakly-supervised 3D Human Pose Estimation
on Human3.6M
(3D Annotations metric)
From Lifestyle Vlogs to Everyday Interactions
A major stumbling block to progress in understanding basic human interactions, such as getting out of bed or opening a refrigerator, is lack of good training data.
Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene
The goal of this paper is to take a single 2D image of a scene and recover the 3D structure in terms of a small set of factors: a layout representing the enclosing surfaces as well as a set of objects represented in terms of shape and pose.
Generic 3D Representation via Pose Estimation and Matching
Though a large body of computer vision research has investigated developing generic semantic representations, efforts towards developing a similar representation for 3D has been limited.
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database.
What Will Happen Next? Forecasting Player Moves in Sports Videos
A large number of very popular team sports involve the act of one team trying to score a goal against the other.
Learning a Multi-View Stereo Machine
We thoroughly evaluate our approach on the ShapeNet dataset and demonstrate the benefits over classical approaches as well as recent learning based methods.
AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1. 58M action labels with multiple labels per person occurring frequently.
Ranked #7 on
Action Detection
on UCF101-24
Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency
We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view.
Hierarchical Surface Prediction for 3D Object Reconstruction
A major limitation of such approaches is that they only predict a coarse resolution voxel grid, which does not capture the surface of the objects well.
Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation
Manipulation of deformable objects, such as ropes and cloth, is an important but challenging problem in robotics.
Learning to Optimize Neural Nets
Learning to Optimize is a recently proposed framework for learning optimization algorithms using reinforcement learning.
Fast k-Nearest Neighbour Search via Prioritized DCI
Most exact methods for k-nearest neighbour search suffer from the curse of dimensionality; that is, their query times exhibit exponential dependence on either the ambient or the intrinsic dimensionality.
Cognitive Mapping and Planning for Visual Navigation
The accumulated belief of the world enables the agent to track visited regions of the environment.
Feedback Networks
Currently, the most successful learning models in computer vision are based on learning successive representations followed by a decision layer.
Beyond Skip Connections: Top-Down Modulation for Object Detection
But most of these fine details are lost in the early convolutional layers.
Ranked #221 on
Object Detection
on COCO test-dev
Learning Shape Abstractions by Assembling Volumetric Primitives
We present a learning framework for abstracting complex shapes by learning to assemble objects using 3D volumetric primitives.
Learning to Poke by Poking: Experiential Learning of Intuitive Physics
We investigate an experiential learning paradigm for acquiring an internal model of intuitive physics.
Learning to Optimize
Algorithm design is a laborious process and often requires many iterations of ideation and validation.
View Synthesis by Appearance Flow
We address the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints.
Amodal Instance Segmentation
We consider the problem of amodal instance segmentation, the objective of which is to predict the region encompassing both visible and occluded parts of each object.
Fast k-Nearest Neighbour Search via Dynamic Continuous Indexing
Existing methods for retrieving k-nearest neighbours suffer from the curse of dimensionality.
Iterative Instance Segmentation
Existing methods for pixel-wise labelling tasks generally disregard the underlying structure of labellings, often leading to predictions that are visually implausible.
Exploring Person Context and Local Scene Context for Object Detection
In this paper we explore two ways of using context for object detection.
Shape and Symmetry Induction for 3D Objects
Actions as simple as grasping an object or navigating around it require a rich understanding of that object's 3D shape from a given viewpoint.
Learning Visual Predictive Models of Physics for Playing Billiards
The ability to plan and execute goal specific actions in varied, unexpected settings is a central requirement of intelligent agents.
Amodal Completion and Size Constancy in Natural Scenes
We consider the problem of enriching current object detection systems with veridical object sizes and relative depth estimates from a single image.
Bandit Label Inference for Weakly Supervised Learning
The scarcity of data annotated at the desired level of granularity is a recurring issue in many applications.
Recurrent Network Models for Human Dynamics
We propose the Encoder-Recurrent-Decoder (ERD) model for recognition and prediction of human body pose in videos and motion capture.
Ranked #8 on
Human Pose Forecasting
on Human3.6M
(MAR, walking, 1,000ms metric)
Human Pose Estimation with Iterative Error Feedback
Hierarchical feature extractors such as Convolutional Networks (ConvNets) have achieved impressive performance on a variety of classification tasks using purely feedforward processing.
Ranked #43 on
Pose Estimation
on MPII Human Pose
Cross Modal Distillation for Supervision Transfer
In this work we propose a technique that transfers supervision between images from different modalities.
Aligning 3D Models to RGB-D Images of Cluttered Scenes
The goal of this work is to represent objects in an RGB-D scene with corresponding 3D models from a library.
Depth From Shading, Defocus, and Correspondence Using Light-Field Angular Coherence
Using shading information is essential to improve the shape estimation.
Visual Semantic Role Labeling
In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction.
DeepBox: Learning Objectness with Convolutional Networks
Existing object proposal approaches use primarily bottom-up cues to rank proposals, while we believe that objectness is in fact a high level construct.
Learning to See by Moving
We show that given the same number of training images, features learnt using egomotion as supervision compare favourably to features learnt using class-label as supervision on visual tasks of scene recognition, object recognition, visual odometry and keypoint matching.
Contextual Action Recognition with R*CNN
In this work, we exploit the simple observation that actions are accompanied by contextual cues to build a strong action recognition system.
Ranked #4 on
Weakly Supervised Object Detection
on HICO-DET
Pose Induction for Novel Object Categories
We address the task of predicting pose for objects of unannotated object categories from a small seed set of annotated object classes.
Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation
We propose a unified approach for bottom-up hierarchical image segmentation and object proposal generation for recognition, called Multiscale Combinatorial Grouping (MCG).
Inferring 3D Object Pose in RGB-D Images
The goal of this work is to replace objects in an RGB-D scene with corresponding 3D models from a library.
Learning to Segment Moving Objects in Videos
We segment moving objects in videos by ranking spatio-temporal segment proposals according to "moving objectness": how likely they are to contain a moving object.
Actions and Attributes from Wholes and Parts
We investigate the importance of parts for the tasks of action and attribute classification.
Grouping-Based Low-Rank Trajectory Completion and 3D Reconstruction
Furthermore, NRSfM needs to be robust to noise in both segmentation and tracking, e. g., drifting, segmentation ``leaking'', optical flow ``bleeding'' etc.
Viewpoints and Keypoints
We characterize the problem of pose estimation for rigid objects in terms of determining viewpoint to explain coarse pose and keypoint prediction to capture the finer details.
Ranked #3 on
Keypoint Detection
on Pascal3D+
