Search Results for author:
Found 63 papers, 26 papers with code
Learning Goal-Conditioned Value Functions with one-step Path rewards rather than Goal-Rewards
Multi-goal reinforcement learning (MGRL) addresses tasks where the desired goal state can change for every trial.
Learning to Estimate External Forces of Human Motion in Video
Analyzing sports performance or preventing injuries requires capturing ground reaction forces (GRFs) exerted by the human body during certain movements.
Q-TART: Quickly Training for Adversarial Robustness and in-Transferability
Raw deep neural network (DNN) performance is not enough; in real-world settings, computational load, training efficiency and adversarial security are just as or even more important.
Evaluating and Improving Interactions with Hazy Oracles
Many AI systems integrate sensor inputs, world knowledge, and human-provided information to perform inference.
The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting
Quantitative evaluation has increased dramatically among recent video inpainting work, but the video and mask content used to gauge performance has received relatively little attention.
Depth from Camera Motion and Object Detection
This paper addresses the problem of learning to estimate the depth of detected objects given some measurement of camera motion (e. g., from robot kinematics or vehicle odometry).
Temporally Guided Articulated Hand Pose Tracking in Surgical Videos
Additionally, we collect the first dataset, Surgical Hands, that provides multi-instance articulated hand pose annotations for in-vivo videos.
Integrating Human Gaze into Attention for Egocentric Activity Recognition
In addition, we model the distribution of gaze fixations using a variational method.
Ranked #1 on
Egocentric Activity Recognition
on EGTEA
The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigation
In each trial, the pair first cooperates to localize the robot on a global map visible to the Commander, then the Driver follows Commander instructions to move the robot to a sequence of target objects.
Ground-truth or DAER: Selective Re-query of Secondary Information
Many vision tasks use secondary information at inference time -- a seed -- to assist a computer vision model in solving a problem.
Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection
We propose an architecture using multimodal convolutional neural networks for fast detection and localization of tools in RAS videos.
Adversarial Background-Aware Loss for Weakly-supervised Temporal Activity Localization
Two triplets of the feature space are considered in our approach: one triplet is used to learn discriminative features for each activity class, and the other one is used to distinguish the features where no activity occurs (i. e. background features) from activity-related features for each video.
Learning Object Depth from Camera Motion and Video Object Segmentation
Video object segmentation, i. e., the separation of a target object from background in video, has made significant progress on real and challenging videos in recent years.
Slimming Neural Networks using Adaptive Connectivity Scores
Finally, we define a novel sensitivity criterion for filters that measures the strength of their contributions to the succeeding layer and highlights critical filters that need to be completely protected from pruning.
Novel Object Viewpoint Estimation through Reconstruction Alignment
Our key insight is that although we do not have an explicit 3D model or a predefined canonical pose, we can still learn to estimate the object's shape in the viewer's frame and then use an image to provide our reference model or canonical pose.
MINT: Deep Network Compression via Mutual Information-based Neuron Trimming
Most approaches to deep neural network compression via pruning either evaluate a filter's importance using its weights or optimize an alternative objective function with sparsity constraints.
Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
In this work, we propose Learning with Incremental Labels and Adaptive Compensation (LILAC), a two-phase method that incrementally increases the number of unique output labels rather than the difficulty of samples while consistently using the entire dataset throughout training.
HyperCon: Image-To-Video Model Transfer for Video-To-Video Translation Tasks
To combine the benefits of image and video models, we propose an image-to-video model transfer method called Hyperconsistency (HyperCon) that transforms any well-trained image model into a temporally consistent video model without fine-tuning.
A Geometric Approach to Online Streaming Feature Selection
In this work, firstly, we assert that OSFS's main assumption of having data from all the samples available at runtime is unrealistic and introduce a new setting where features and samples are streamed concurrently called OSFS with Streaming Samples (OSFS-SS).
Unified Vision-Language Pre-Training for Image Captioning and VQA
The model is unified in that (1) it can be fine-tuned for either vision-language generation (e. g., image captioning) or understanding (e. g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models.
Ranked #1 on
Image Captioning
on Flickr30k Captions test
TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection
It consists of two building blocks: first, the encoder network extracts low-resolution spatiotemporal features from an input clip of several consecutive frames, and then the following prediction network decodes the encoded features spatially while aggregating all the temporal information.
Ranked #3 on
Video Saliency Detection
on DHF1K
Multi-Channel Attention Selection GAN with Cascaded Semantic Guidance for Cross-View Image Translation
In this paper, we propose a novel approach named Multi-Channel Attention SelectionGAN (SelectionGAN) that makes it possible to generate images of natural scenes in arbitrary viewpoints, based on an image of the scene and a novel semantic map.
Bird View Synthesis
Cross-View Image-to-Image Translation
+1
BubbleNets: Learning to Select the Guidance Frame in Video Object Segmentation by Deep Sorting Frames
Semi-supervised video object segmentation has made significant progress on real and challenging videos in recent years.
Semantic Segmentation
Semi-Supervised Video Object Segmentation
+1
Video Object Segmentation-based Visual Servo Control and Object Depth Estimation on a Mobile Robot Platform
To be useful in everyday environments, robots must be able to identify and locate unstructured, real-world objects.
Robotics
Attribute-Guided Sketch Generation
To this end, we propose a novel Attribute-Guided Sketch Generative Adversarial Network (ASGAN) which is an end-to-end framework and contains two pairs of generators and discriminators, one of which is used to generate faces with attributes while the other one is employed for image-to-sketch translation.
Grounded Video Description
Our dataset, ActivityNet-Entities, augments the challenging ActivityNet Captions dataset with 158k bounding box annotations, each grounding a noun phrase.
Dynamic Graph Modules for Modeling Object-Object Interactions in Activity Recognition
Video action recognition, a critical problem in video understanding, has been gaining increasing attention.
Tukey-Inspired Video Object Segmentation
We investigate the problem of strictly unsupervised video object segmentation, i. e., the separation of a primary object from background in video without a user-provided object mask or any training on an annotated dataset.
Semantic Segmentation
Unsupervised Video Object Segmentation
+1
Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals
We do this by adapting the Floyd-Warshall algorithm for RL and call the adaptation Floyd-Warshall RL (FWRL).
Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
A naive extension of this approach to the video domain is to treat the entire segment as a bag of spatial object proposals.
Joint Surgical Gesture and Task Classification with Multi-Task and Multimodal Learning
Our experimental results show that our approach is superior compared to an ar- chitecture that classifies the gestures and surgical tasks separately on visual cues and motion cues respectively.
M-PACT: An Open Source Platform for Repeatable Activity Classification Research
We present M-PACT to overcome existing issues by removing the need to develop boilerplate code which allows users to quickly prototype action classification models while leveraging existing state-of-the-art (SOTA) models available in the platform.
End-to-End Dense Video Captioning with Masked Transformer
To address this problem, we propose an end-to-end transformer model for dense video captioning.
Ranked #7 on
Video Captioning
on YouCook2
Learning Kinematic Descriptions using SPARE: Simulated and Physical ARticulated Extendable dataset
SPARE is an extendable open-source dataset providing equivalent simulated and physical instances of articulated objects (kinematic chains), providing the greater research community with a training and evaluation tool for methods generating kinematic descriptions of articulated objects.
T-RECS: Training for Rate-Invariant Embeddings by Controlling Speed for Action Recognition
We explore the erratic behavior caused by this phenomena on state-of-the-art deep network-based methods for action recognition in terms of maximum performance and stability in recognition accuracy across a range of input video speeds.
A Temporally-Aware Interpolation Network for Video Frame Inpainting
We propose the first deep learning solution to video frame inpainting, a challenging instance of the general video inpainting problem with applications in video editing, manipulation, and forensics.
A Dataset To Evaluate The Representations Learned By Video Prediction Models
We present a parameterized synthetic dataset called Moving Symbols to support the objective study of video prediction networks.
A Critical Investigation of Deep Reinforcement Learning for Navigation
However, when trained and tested on different sets of maps, the algorithm fails to transfer the ability to gather and exploit map-information to unseen maps.
Adviser Networks: Learning What Question to Ask for Human-In-The-Loop Viewpoint Estimation
We address this question by formulating it as an Adviser Problem: can we learn a mapping from the input to a specific question to ask the human to maximize the expected positive impact to the overall task?
Deep Net Triage: Analyzing the Importance of Network Layers via Structural Compression
As such, while certain structures have been found to work better than others, the significance of a model's unique structure, or the importance of a given layer, and how these translate to overall accuracy, remains unclear.
Predicting Future Lane Changes of Other Highway Vehicles using RNN-based Deep Models
To demonstrate our approach, we validate our model using authentic interstate highway driving to predict the future lane change maneuvers of other vehicles neighboring our autonomous vehicle.
Do Deep Reinforcement Learning Algorithms really Learn to Navigate?
As the title of the paper by Mirowski et al. (2016) suggests, one might assume that DRL-based algorithms are able to “learn to navigate” and are thus ready to replace classical mapping and path-planning algorithms, at least in simulated environments.
Weakly Supervised Actor-Action Segmentation via Robust Multi-Task Ranking
However, current methods for detailed understanding of actor and action have significant limitations: they require large amounts of finely labeled data, and they fail to capture any internal relationship among actors and actions.
Action Understanding with Multiple Classes of Actors
Despite the rapid progress, existing works on action understanding focus strictly on one type of action agent, which we call actor---a human adult, ignoring the diversity of actions performed by other actors.
Video Object Segmentation using Supervoxel-Based Gerrymandering
Focusing on the problem of strictly unsupervised video object segmentation, we devise a method called supervoxel gerrymandering that links masks of foregroundness and backgroundness via local and non-local consensus measures.
Click Here: Human-Localized Keypoints as Guidance for Viewpoint Estimation
We motivate and address a human-in-the-loop variant of the monocular viewpoint estimation task in which the location and class of one semantic object keypoint is available at test time.
Towards Automatic Learning of Procedures from Web Instructional Videos
To answer this question, we introduce the problem of procedure segmentation--to segment a video procedure into category-independent procedure segments.
Sparse Factorization Layers for Neural Networks with Limited Supervision
On the other hand, dictionary learning does not scale to the size of problems that CNNs can handle, despite being very effective at low-level vision tasks such as denoising and inpainting.
Watch What You Just Said: Image Captioning with Text-Conditional Attention
Attention mechanisms have attracted considerable interest in image captioning due to its powerful performance.
A Continuous Occlusion Model for Road Scene Understanding
We present a physically interpretable, continuous 3D model for handling occlusions with applications to road scene understanding.
Spatiotemporal Articulated Models for Dynamic SLAM
We propose an online spatiotemporal articulation model estimation framework that estimates both articulated structure as well as a temporal prediction model solely using passive observations.
LIBSVX: A Supervoxel Library and Benchmark for Early Video Processing
Supervoxel segmentation has strong potential to be incorporated into early video analysis as superpixel segmentation has in image analysis.
Actor-Action Semantic Segmentation with Grouping Process Models
Actor-action semantic segmentation made an important step toward advanced video understanding problems: what action is happening; who is performing the action; and where is the action in space-time.
Action Detection by Implicit Intentional Motion Clustering
This paper hence seeks to understand the spatiotemporal properties of intentional movement and how to capture such intentional movement without relying on challenging human detection and tracking.
Human Action Segmentation With Hierarchical Supervoxel Consistency
Detailed analysis of human action, such as action classification, detection and localization has received increasing attention from the community; datasets like JHMDB have made it plausible to conduct studies analyzing the impact that such deeper information has on the greater action understanding problem.
Can Humans Fly? Action Understanding With Multiple Classes of Actors
There is no work we know of on simultaneously inferring actors and actions in the video, not to mention a dataset to experiment with.
Compositional Structure Learning for Action Understanding
The focus of the action understanding literature has predominately been classification, how- ever, there are many applications demanding richer action understanding such as mobile robotics and video search, with solutions to classification, localization and detection.
Actionness Ranking with Lattice Conditional Ordinal Random Fields
Action analysis in image and video has been attracting more and more attention in computer vision.
Semi-Supervised Nonlinear Distance Metric Learning via Forests of Max-Margin Cluster Hierarchies
By introducing randomness during hierarchy training and combining the output of many of the resulting semi-random weak hierarchy metrics, we can obtain a powerful and robust nonlinear metric model.
Active Clustering with Model-Based Uncertainty Reduction
Semi-supervised clustering seeks to augment traditional clustering methods by incorporating side information provided via human expertise in order to increase the semantic meaningfulness of the resulting clusters.
A Study of Actor and Action Semantic Retention in Video Supervoxel Segmentation
In this paper, we conduct a systematic study of how well the actor and action semantics are retained in video supervoxel segmentation.
A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching
The problem of describing images through natural language has gained importance in the computer vision community.
Hamiltonian Streamline Guided Feature Extraction with Applications to Face Detection
We propose a new feature extraction method based on two dynamical systems induced by intensity landscape: the negative gradient system and the Hamiltonian system.
