Search Results for author:
Found 184 papers, 68 papers with code
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition
We analyze the suitability of our new primitive for video action recognition and explore several novel variations of our approach to enable stronger representational flexibility while maintaining an efficient design.
Holistic Evaluation of Text-To-Image Models
The stunning qualitative improvement of recent text-to-image models has led to their widespread attention and adoption.
NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities
We present Neural Signal Operated Intelligent Robots (NOIR), a general-purpose, intelligent brain-robot interface system that enables humans to command robots to perform everyday activities through brain signals.
ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image
Further, we observe that Score Distillation Sampling (SDS) tends to truncate the distribution of complex backgrounds during distillation of 360-degree scenes, and propose "SDS anchoring" to improve the diversity of synthesized novel views.
Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI
We present Mini-BEHAVIOR, a novel benchmark for embodied AI that challenges agents to use reasoning and decision-making skills to solve complex activities that resemble everyday human challenges.
D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Robotic Manipulation
These fields capture the dynamics of the underlying 3D environment and encode both semantic features and instance masks.
MindAgent: Emergent Gaming Interaction
Large Language Models (LLMs) have the capacity of performing complex scheduling in a multi-agent system and can coordinate these agents into completing sophisticated tasks that require extensive collaboration.
Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation
However, the challenges arise due to the high-dimensional action space of dexterous hand and complex compositional dynamics of the long-horizon tasks.
Rendering Humans from Object-Occluded Monocular Videos
3D understanding and rendering of moving humans from monocular videos is a challenging task.
Primitive Skill-based Robot Learning from Human Evaluative Feedback
By combining them, SEED reduces the human effort required in RLHF and increases safety in training robot manipulation with RL in real-world settings.
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations.
Dynamic-Resolution Model Learning for Object Pile Manipulation
Prior works typically assume representation at a fixed dimension or resolution, which may be inefficient for simple tasks and ineffective for more complicated tasks.
Differentially Private Video Activity Recognition
In recent years, differential privacy has seen significant advancements in image classification; however, its application to video activity recognition remains under-explored.
Task-Driven Graph Attention for Hierarchical Relational Object Navigation
Embodied AI agents in large scenes often need to navigate to find objects.
HomE: Homography-Equivariant Video Representation Learning
In this work, we propose a novel method for representation learning of multi-view videos, where we explicitly model the representation space to maintain Homography Equivariance (HomE).
The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects
We introduce the ObjectFolder Benchmark, a benchmark suite of 10 tasks for multisensory object-centric learning, centered around object recognition, reconstruction, and manipulation with sight, sound, and touch.
Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear
We introduce Sonicverse, a multisensory simulation platform with integrated audio-visual simulation for training household agents that can both see and hear.
Modeling Dynamic Environments with Scene Graph Memory
We evaluate our method in the Dynamic House Simulator, a new benchmark that creates diverse dynamic graphs following the semantic patterns typically seen at homes, and show that NEP can be trained to predict the locations of objects in a variety of environments with diverse object movement dynamics, outperforming baselines both in terms of new scene adaptability and overall accuracy.
See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation
Humans use all of their senses to accomplish different tasks in everyday activities.
Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Tasks
ATR selects suitable tasks, which consist of an initial environment state and manipulation goal, for learning robust skills by balancing the diversity and feasibility of the tasks.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Reward
These results identify tasks where expectation alignment is a more useful strategy than curiosity-driven exploration for multi-agent coordination, enabling agents to do zero-shot coordination.
VIMA: General Robot Manipulation with Multimodal Prompts
We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens.
GaitForeMer: Self-Supervised Pre-Training of Transformers via Human Motion Forecasting for Few-Shot Gait Impairment Severity Estimation
Because of this clinical data scarcity and inspired by the recent advances in self-supervised large-scale language models like GPT-3, we use human motion forecasting as an effective self-supervised pre-training task for the estimation of motor impairment severity.
MaskViT: Masked Visual Pre-Training for Video Prediction
This work shows that we can create good video prediction models by pre-training transformers via masked visual modeling.
BEHAVIOR in Habitat 2.0: Simulator-Independent Logical Task Description for Benchmarking Embodied AI Agents
Robots excel in performing repetitive and precision-sensitive tasks in controlled environments such as warehouses and factories, but have not been yet extended to embodied AI agents providing assistance in household tasks.
PrivHAR: Recognizing Human Actions From Privacy-preserving Lens
The accelerated use of digital cameras prompts an increasing concern about privacy and security, particularly in applications such as action recognition.
Revisiting the "Video" in Video-Language Understanding
Building on recent progress in self-supervised image-language models, we revisit this question in the context of video and language tasks.
Ranked #1 on
Video Question Answering
on MSR-VTT-MC
ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer
We present ObjectFolder 2. 0, a large-scale, multisensory dataset of common household objects in the form of implicit neural representations that significantly enhances ObjectFolder 1. 0 in three aspects.
MetaMorph: Learning Universal Controllers with Transformers
Multiple domains like vision, natural language, and audio are witnessing tremendous progress by leveraging Transformers for large scale pre-training followed by task specific fine tuning.
Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation
Doing this is challenging for two reasons: on the data side, current interfaces make collecting high-quality human demonstrations difficult, and on the learning side, policies trained on limited data can suffer from covariate shift when deployed.
Visual Intelligence through Human Interaction
Over the last decade, Computer Vision, the branch of Artificial Intelligence aimed at understanding the visual world, has evolved from simply recognizing objects in images to describing pictures, answering questions about images, aiding robots maneuver around physical spaces and even generating novel visual content.
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
In this paper, we study the problem of learning a repertoire of low-level skills from raw images that can be sequenced to complete long-horizon visuomotor tasks.
Model-based Reinforcement Learning
reinforcement-learning
+1
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
Multisensory object-centric perception, reasoning, and interaction have been a key research topic in recent years.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
Our method co-optimizes a human policy and a robot policy in an interactive learning process: the human policy learns to generate diverse and plausible collaborative behaviors from demonstrations while the robot policy learns to assist by estimating the unobserved latent strategy of its human collaborator.
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation.
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
We evaluate the new capabilities of iGibson 2. 0 to enable robot learning of novel tasks, in the hope of demonstrating the potential of this new simulator to support new research in embodied AI.
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation.
Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning
Although virtual agents are increasingly situated in environments where natural language is the most effective mode of interaction with humans, these exchanges are rarely used as an opportunity for learning.
Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering
Active learning promises to alleviate the massive data needs of supervised machine learning: it has successfully improved sample efficiency by an order of magnitude on traditional tasks like topic classification and object recognition.
Discovering Generalizable Skills via Automated Generation of Diverse Tasks
To encourage generalizable skills to emerge, our method trains each skill to specialize in the paired task and maximizes the diversity of the generated tasks.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Scalable Differential Privacy With Sparse Network Finetuning
We propose a novel method for privacy-preserving training of deep neural networks leveraging public, out-domain data.
SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
A student network then learns to mimic the expert policy by supervised learning with strong augmentations, making its representation more robust against visual variations compared to the expert.
Physion: Evaluating Physical Prediction from Vision in Humans and Machines
While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments.
Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution.
Metadata Normalization
Batch Normalization (BN) and its variants have delivered tremendous success in combating the covariate shift induced by the training step of deep learning methods.
A Study of Face Obfuscation in ImageNet
In this paper, we explore the effects of face obfuscation on the popular ImageNet challenge visual recognition benchmark.
Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
Our key insight is that greedy and modular optimization of hierarchical autoencoders can simultaneously address both the memory constraints and the optimization challenges of large-scale video prediction.
Ranked #1 on
Video Prediction
on Cityscapes 128x128
Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control
Key to such capability is hand-eye coordination, a cognitive ability that enables humans to adaptively direct their movements at task-relevant objects and be invariant to the objects' absolute spatial location.
Embodied Intelligence via Learning and Evolution
However, the principles governing relations between environmental complexity, evolved morphology, and the learnability of intelligent control, remain elusive, partially due to the substantial challenge of performing large-scale in silico experiments on evolution and learning.
Human-in-the-Loop Imitation Learning using Remote Teleoperation
We develop a simple and effective algorithm to train the policy iteratively on new data collected by the system that encourages the policy to learn how to traverse bottlenecks through the interventions.
Learning Multi-Arm Manipulation Through Collaborative Teleoperation
To address these challenges, we present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks.
IGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
We present iGibson 1. 0, a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes.
Conceptual Metaphors Impact Perceptions of Human-AI Collaboration
In a third study, we assess effects of metaphor choices on potential users' desire to try out the system and find that users are drawn to systems that project higher competence and warmth.
Vision-based Estimation of MDS-UPDRS Gait Scores for Assessing Parkinson's Disease Motor Severity
This is the first benchmark for classifying PD patients based on MDS-UPDRS gait severity and could be an objective biomarker for disease severity.
Adaptive Procedural Task Generation for Hard-Exploration Problems
To enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balancing the agent's performance in the generated tasks and the similarity to the target tasks.
Learning Physical Graph Representations from Visual Scenes
To overcome these limitations, we introduce the idea of Physical Scene Graphs (PSGs), which represent scenes as hierarchical graphs, with nodes in the hierarchy corresponding intuitively to object parts at different scales, and edges to physical connections between parts.
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
In the second stage of GTI, we collect a small set of rollouts from the unconditioned stochastic policy of the first stage, and train a goal-directed agent to generalize to novel start and goal configurations.
Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy
Computer vision technology is being used by many but remains representative of only a few.
Action Genome: Actions as Composition of Spatio-temporal Scene Graphs
Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42. 7% mAP using as few as 10 examples.
Deep Bayesian Active Learning for Multiple Correct Outputs
The assumption that these tasks always have exactly one correct answer has resulted in the creation of numerous uncertainty-based measurements, such as entropy and least confidence, which operate over a model's outputs.
Motion Reasoning for Goal-Based Imitation Learning
We further show that by using the automatically inferred goal from the video demonstration, our robot is able to reproduce the same task in a real kitchen environment.
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data
For simple short-horizon manipulation tasks with modest variation in task instances, offline learning from a small set of demonstrations can produce controllers that successfully solve the task.
Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity
We evaluate the quality of our platform, the diversity of demonstrations in our dataset, and the utility of our dataset via quantitative and qualitative analysis.
Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments
We present Interactive Gibson Benchmark, the first comprehensive benchmark for training and evaluating Interactive Navigation: robot navigation strategies where physical interaction with objects is allowed and even encouraged to accomplish a task.
Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
The fundamental challenge of planning for multi-step manipulation is to find effective and plausible action sequences that lead to the task goal.
KETO: Learning Keypoint Representations for Tool Manipulation
For this purpose, we present KETO, a framework of learning keypoint representations of tool-based manipulation.
Robotics
6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints
We present 6-PACK, a deep learning approach to category-level 6D object pose tracking on RGB-D data.
Ranked #1 on
6D Pose Estimation using RGBD
on REAL275
(Rerr metric)
Representation Learning with Statistical Independence to Mitigate Bias
Presence of bias (in datasets or tasks) is inarguably one of the most critical challenges in machine learning applications that has alluded to pivotal debates in recent years.
Causal Induction from Visual Observations for Goal Directed Tasks
Causal reasoning has been an indispensable capability for humans and other intelligent animals to interact with the physical world.
Regression Planning Networks
Recent learning-to-plan methods have shown promising results on planning directly from observation space.
DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs
A major difficulty of solving continuous POMDPs is to infer the multi-modal distribution of the unobserved true states and to make the planning algorithm dependent on the perceived uncertainty.
SURREAL-System: Fully-Integrated Stack for Distributed Deep Reinforcement Learning
We present an overview of SURREAL-System, a reproducible, flexible, and scalable framework for distributed reinforcement learning (RL).
Stochastic Neural Physics Predictor
Recently, neural-network based forward dynamics models have been proposed that attempt to learn the dynamics of physical systems in a deterministic way.
Situational Fusion of Visual Representation for Visual Navigation
A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities.
Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning
The key technical challenge is that the symbol grounding is prone to error with limited training data and leads to subsequent symbolic planning failures.
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback.
Procedure Planning in Instructional Videos
In this paper, we study the problem of procedure planning in instructional videos, which can be seen as a step towards enabling autonomous agents to plan for complex tasks in everyday settings such as cooking.
Learning Predicates as Functions to Enable Few-shot Scene Graph Prediction
We introduce the first scene graph prediction model that supports few-shot learning of predicates.
Eidetic 3D LSTM: A Model for Video Prediction and Beyond
We first evaluate the E3D-LSTM network on widely-used future video prediction datasets and achieve the state-of-the-art performance.
Ranked #1 on
Video Prediction
on KTH
(Cond metric)
Scene Graph Prediction with Limited Labels
All scene graph models to date are limited to training on a small set of visual relationships that have thousands of training labels each.
Ranked #1 on
Scene Graph Detection
on VRD
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
We construct Human eYe Perceptual Evaluation (HYPE) a human benchmark that is (1) grounded in psychophysics research in perception, (2) reliable across different sets of randomly sampled outputs from a model, (3) able to produce separable model performances, and (4) efficient in cost and time.
Information Maximizing Visual Question Generation
We build a model that maximizes mutual information between the image, the expected answer and the generated question.
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
Many robotic applications require the agent to perform long-horizon tasks in partially observable environments.
Audio-Linguistic Embeddings for Spoken Sentences
We propose spoken sentence embeddings which capture both acoustic and linguistic content.
Peeking into the Future: Predicting Future Person Activities and Locations in Videos
To facilitate the training, the network is learned with an auxiliary task of predicting future location in which the activity will happen.
Ranked #1 on
Activity Prediction
on ActEV
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources.
Ranked #4 on
6D Pose Estimation
on LineMOD
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
Therefore, we propose to search the network level structure in addition to the cell level structure, which forms a hierarchical architecture search space.
Ranked #7 on
Semantic Segmentation
on PASCAL VOC 2012 val
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation
The key technical challenge for discriminative modeling with weak supervision is that the loss function of the ordering supervision is usually formulated using dynamic programming and is thus not differentiable.
Composing Text and Image for Image Retrieval - An Empirical Odyssey
In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image.
Ranked #2 on
Image Retrieval with Multi-Modal Query
on MIT-States
Vision-Based Gait Analysis for Senior Care
As the senior population rapidly increases, it is challenging yet crucial to provide effective long-term care for seniors who live at home or in senior care facilities.
Privacy-Preserving Action Recognition for Smart Hospitals using Low-Resolution Depth Images
Computer-vision hospital systems can greatly assist healthcare workers and improve medical facility treatment, but often face patient resistance due to the perceived intrusiveness and violation of privacy associated with visual surveillance.
Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference
Homomorphic encryption enables arbitrary computation over data while it remains encrypted.
Cryptography and Security
Measuring Depression Symptom Severity from Spoken Language and 3D Facial Expressions
This technology could be deployed to cell phones worldwide and facilitate low-cost universal access to mental health care.
RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
Imitation Learning has empowered recent advances in learning robotic manipulation tasks by addressing shortcomings of Reinforcement Learning such as exploration and reward specification.
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback.
Temporal Modular Networks for Retrieving Complex Compositional Activities in Videos
A major challenge in computer vision is scaling activity understanding to the long tail of complex activities without requiring collecting large quantities of data for new actions.
Neural Graph Matching Networks for Fewshot 3D Action Recognition
We propose Neural Graph Matching (NGM) Networks, a novel framework that can learn to recognize a previous unseen 3D action class with only a few examples.
Ranked #1 on
Skeleton Based Action Recognition
on CAD-120
Dynamic Task Prioritization for Multitask Learning
We propose dynamic task prioritization for multitask learning.
HiDDeN: Hiding Data With Deep Networks
We show that these encodings are competitive with existing data hiding algorithms, and further that they can be made robust to noise: our models learn to reconstruct hidden information in an encoded image despite the presence of Gaussian blurring, pixel-wise dropout, cropping, and JPEG compression.
Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model.
Distributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go?
One of the most widely used optimization methods for large-scale machine learning problems is distributed asynchronous stochastic gradient descent (DASGD).
Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision
We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering.
Flexible Neural Representation for Physics Prediction
Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail.
Learning to Decompose and Disentangle Representations for Video Prediction
Our goal is to predict future video frames given a sequence of input frames.
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos
In this work, we propose to tackle this new task with a weakly-supervised framework for reference-aware visual grounding in instructional videos, where only the temporal alignment between the transcription and the video segment are available for supervision.
What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets
The ability to capture temporal information has been critical to the development of video understanding models.
Image Generation from Scene Graphs
To overcome this limitation we propose a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships.
Ranked #4 on
Layout-to-Image Generation
on Visual Genome 64x64
Image Generation from Scene Graphs
Layout-to-Image Generation
Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks
Understanding human motion behavior is critical for autonomous moving platforms (like self-driving cars and social robots) if they are to navigate human-centric environments.
Ranked #4 on
Trajectory Prediction
on ETH
Iterative Visual Reasoning Beyond Convolutions
The framework consists of two core modules: a local module that uses spatial memory to store previous beliefs with parallel updates; and a global graph-reasoning module.
Referring Relationships
We formulate the cyclic condition between the entities in a relationship by modelling predicates that connect the entities as shifts in attention from one entity to another.
Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks
We show that our method both effectively detects the spatial bounds of tools as well as significantly outperforms existing methods on tool presence detection.
Learning to Play with Intrinsically-Motivated Self-Aware Agents
We demonstrate that this policy causes the agent to explore novel and informative interactions with its environment, leading to the generation of a spectrum of complex behaviors, including ego-motion prediction, object attention, and object gathering.
Emergence of Structured Behaviors from Curiosity-Based Intrinsic Motivation
Moreover, the world model that the agent learns supports improved performance on object dynamics prediction and localization tasks.
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
Recent deep networks are capable of memorizing the entire data even when the labels are completely random.
Ranked #16 on
Image Classification
on WebVision-1000
Progressive Neural Architecture Search
We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms.
Ranked #14 on
Neural Architecture Search
on NAS-Bench-201, ImageNet-16-120
(Accuracy (Val) metric)
Graph Distillation for Action Detection with Privileged Modalities
We propose a technique that tackles action detection in multimodal videos under a realistic and challenging condition in which only limited training data and partially observed modalities are available.
Label Efficient Learning of Transferable Representations across Domains and Tasks
We propose a framework that learns a representation transferable across different domains and tasks in a label efficient manner.
Thoracic Disease Identification and Localization with Limited Supervision
Accurate identification and localization of abnormalities from radiology images play an integral part in clinical diagnosis and treatment planning.
Neural Task Programming: Learning to Generalize Across Hierarchical Tasks
In this work, we propose a novel robot learning framework called Neural Task Programming (NTP), which bridges the idea of few-shot learning from demonstration and neural program induction.
Scalable Annotation of Fine-Grained Categories Without Experts
We present a crowdsourcing workflow to collect image annotations for visually similar synthetic categories without requiring experts.
Fine-Grained Car Detection for Visual Census Estimation
In this work, we leverage the ubiquity of Google Street View images and develop a computer vision pipeline to predict income, per capita carbon emission, crime rates and other city attributes from a single source of publicly available visual data.
Fine-grained Recognition in the Wild: A Multi-Task Domain Adaptation Approach
While fine-grained object recognition is an important problem in computer vision, current models are unlikely to accurately classify objects in the wild.
Towards Vision-Based Smart Hospitals: A System for Tracking and Monitoring Hand Hygiene Compliance
One in twenty-five patients admitted to a hospital will suffer from a hospital acquired infection.
Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals
Physiological signals such as heart rate can provide valuable information about an individual's state and activity.
Knowledge Acquisition for Visual Question Answering via Iterative Querying
Humans possess an extraordinary ability to learn new skills and new knowledge for problem solving.
Tackling Over-pruning in Variational Autoencoders
The so-called epitomes of this model are groups of mutually exclusive latent factors that compete to explain the data.
Learning to Learn from Noisy Web Videos
Our method uses Q-learning to learn a data labeling policy on a small labeled training dataset, and then uses this to automatically label noisy web data for new visual concepts.
Visual Semantic Planning using Deep Successor Representations
A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world.
Inferring and Executing Programs for Visual Reasoning
Existing methods for visual reasoning attempt to directly map inputs to outputs using black-box architectures without explicitly modeling the underlying reasoning processes.
Ranked #5 on
Visual Question Answering (VQA)
on CLEVR-Humans
Characterizing and Improving Stability in Neural Style Transfer
Recent progress in style transfer on images has focused on improving the quality of stylized images and speed of methods.
Dense-Captioning Events in Videos
We also introduce ActivityNet Captions, a large-scale benchmark for dense-captioning events.
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
We propose an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity (e. g., "dressing") to the action (e. g., "mix yogurt") that produced it.
Using Deep Learning and Google Street View to Estimate the Demographic Makeup of the US
The United States spends more than $1B each year on initiatives such as the American Community Survey (ACS), a labor-intensive door-to-door study that measures statistics relating to race, gender, education, occupation, unemployment, and other demographic factors.
Scene Graph Generation by Iterative Message Passing
In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image.
Ranked #7 on
Panoptic Scene Graph Generation
on PSG Dataset
Unsupervised Learning of Long-Term Motion Dynamics for Videos
We present an unsupervised representation learning approach that compactly encodes the motion dependencies in videos.
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings.
Recurrent Attention Models for Depth-Based Person Identification
We present an attention-based model that reasons on human body shape and motion dynamics to identify individuals in the absence of RGB information, hence in the dark.
A Hierarchical Approach for Generating Descriptive Image Paragraphs
Recent progress on image captioning has made it possible to generate novel sentences describing images in natural language, but compressing an image into a single sentence can describe visual content in only coarse detail.
Crowdsourcing in Computer Vision
Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts.
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine.
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality
Microtask crowdsourcing is increasingly critical to the creation of extremely large datasets.
Visual Relationship Detection with Language Priors
We improve on prior work by leveraging language priors from semantic word embeddings to finetune the likelihood of a predicted relationship.
Ranked #2 on
Scene Graph Generation
on VRD
Connectionist Temporal Modeling for Weakly Supervised Action Labeling
We propose a weakly-supervised framework for action labeling in video, where only the order of occurring actions is required during training time.
Locally-Optimized Inter-Subject Alignment of Functional Cortical Regions
Our method outperforms the two most commonly used alternatives (anatomical landmark-based AFNI alignment and cortical convexity-based FreeSurfer alignment) in overlap between predicted region and functionally-defined LOC.
Social LSTM: Human Trajectory Prediction in Crowded Spaces
Different from the conventional LSTM, we share the information between multiple LSTMs through a new pooling layer.
Ranked #1 on
Trajectory Prediction
on Stanford Drone
(ADE (8/12) @K=5 metric)
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
We consider image transformation problems, where an input image is transformed into an output image.
Ranked #4 on
Nuclear Segmentation
on Cell17
Towards Viewpoint Invariant 3D Human Pose Estimation
We propose a viewpoint invariant model for 3D human pose estimation from a single depth image.
Ranked #4 on
Pose Estimation
on ITOP top-view
Embracing Error to Enable Rapid Crowdsourcing
Microtask crowdsourcing has enabled dataset advances in social science and machine learning, but existing crowdsourcing schemes are too expensive to scale up with the expanding volume of data.
RGB-W: When Vision Meets Wireless
Inspired by the recent success of RGB-D cameras, we propose the enrichment of RGB data with an additional "quasi-free" modality, namely, the wireless signal (e. g., wifi or Bluetooth) emitted by individuals' cell phones, referred to as RGB-W.
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
We introduce the dense captioning task, which requires a computer vision system to both localize and describe salient regions in images in natural language.
Ranked #3 on
Dense Captioning
on Visual Genome
End-to-end Learning of Action Detection from Frame Glimpses in Videos
In this work we introduce a fully end-to-end approach for action detection in videos that learns to directly predict the temporal bounds of actions.
Ranked #9 on
Temporal Action Localization
on THUMOS’14
(mAP IOU@0.2 metric)
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition
Current approaches for fine-grained recognition do the following: First, recruit experts to annotate a dataset of images, optionally also collecting more structured data in the form of part annotations and bounding boxes.
Ranked #4 on
Fine-Grained Image Classification
on CUB-200-2011
(using extra training data)
Visual7W: Grounded Question Answering in Images
It enables a new type of QA with visual answers, in addition to textual answers used in previous work.
Multiple-choice
Multiple Choice Question Answering (MCQA)
+2
Detecting events and key actors in multi-person videos
In this paper, we propose a model which learns to detect events in such videos while automatically "attending" to the people responsible for the event.
Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
Every moment counts in action recognition.
Ranked #7 on
Action Detection
on Multi-THUMOS