Search Results for author:
Found 47 papers, 12 papers with code
Assessing the State of Self-Supervised Human Activity Recognition using Wearables
As such, self-supervision, i. e., the paradigm of 'pretrain-then-finetune' has the potential to become a strong alternative to the predominant end-to-end training approaches, let alone the classic activity recognition chain with hand-crafted features of sensor data.
Learning Temporal Rules from Noisy Timeseries Data
Events across a timeline are a common data representation, seen in different temporal modalities.
BLT: Bidirectional Layout Transformer for Controllable Layout Generation
Our results demonstrate two key advances to the state-of-the-art layout transformer models.
VideoPose: Estimating 6D object pose from videos
We introduce a simple yet effective algorithm that uses convolutional neural networks to directly estimate object poses from videos.
Discrete Representations Strengthen Vision Transformer Robustness
Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition.
Ranked #1 on
Domain Generalization
on Stylized-ImageNet
Neural Temporal Logic Programming
Events across a timeline are a common data representation, seen in different temporal modalities.
Unsupervised Discovery of Actions in Instructional Videos
In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos.
Unsupervised Action Segmentation for Instructional Videos
In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos, which are rarely annotated with atomic actions.
Automatic Non-Linear Video Editing Transfer
We propose an automatic approach that extracts editing styles in a source video and applies the edits to matched footage for video creation.
PLAN-B: Predicting Likely Alternative Next Best Sequences for Action Prediction
Action prediction focuses on anticipating actions before they happen.
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget
PointGoal navigation has seen significant recent interest and progress, spurred on by the Habitat platform and associated challenge.
Contrastive Predictive Coding for Human Activity Recognition
Our work focuses on effective use of small amounts of labeled data and the opportunistic exploitation of unlabeled data that are straightforward to collect in mobile and ubiquitous computing scenarios.
Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views
We study the task of semantic mapping - specifically, an embodied agent (a robot or an egocentric AI assistant) is given a tour of a new environment and asked to build an allocentric top-down semantic map ("what is where?")
Text as Neural Operator: Image Manipulation by Text Instruction
In recent years, text-guided image manipulation has gained increasing attention in the multimedia and computer vision community.
Analyzing Visual Representations in Embodied Navigation Tasks
Recent advances in deep reinforcement learning require a large amount of training data and generally result in representations that are often over specialized to the target task.
Insights on Visual Representations for Embodied Navigation Tasks
Surprisingly, we find that slight differences in task have no measurable effect on the visual representation for both SqueezeNet and ResNet architectures.
Neural Design Network: Graphic Layout Generation with Constraints
The first module predicts a graph with complete relations from a graph with user-specified relations.
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
We leverage this scaling to train an agent for 2. 5 Billion steps of experience (the equivalent of 80 years of human experience) -- over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs.
Ranked #1 on
PointGoal Navigation
on Gibson PointGoal Navigation
Estimating Mass Distribution of Articulated Objects using Non-prehensile Manipulation
Using our method, we train a robotic arm to estimate the mass distribution of an object with moving parts (e. g. an articulated rigid body system) by pushing it on a surface with unknown friction properties.
Novel evaluation of surgical activity recognition models using task-based efficiency metrics
Conclusions: We demonstrate that metrics-based evaluation of surgical activity recognition models is a viable approach to determine when models can be used to quantify surgical efficiencies.
Floors are Flat: Leveraging Semantics for Real-Time Surface Normal Prediction
We propose 4 insights that help to significantly improve the performance of deep learning models that predict surface normals and semantic labels from a single RGB image.
Ranked #1 on
Surface Normals Estimation
on ScanNetV2
Embodied Question Answering in Photorealistic Environments with Point Cloud Perception
To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task -- Embodied Question Answering [1] in photo-realistic environments (Matterport 3D).
Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer
We show that pretraining real-time segmentation architectures with synthetic segmentation data instead of ImageNet improves fine-tuning performance by reducing the bias learned in pretraining and closing the \textit{transfer gap} as a result.
Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition
We propose a self-supervised learning method to jointly reason about spatial and temporal context for video recognition.
End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
We introduce a new dataset of dialogs about videos of human behaviors.
Surgical Activity Recognition in Robot-Assisted Radical Prostatectomy using Deep Learning
Adverse surgical outcomes are costly to patients and hospitals.
Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7
Scene-aware dialog systems will be able to have conversations with users about the objects and events around them.
Object category learning and retrieval with weak supervision
We consider the problem of retrieving objects from image data and learning to classify them into meaningful semantic categories with minimal supervision.
Efficient Hierarchical Graph-Based Segmentation of RGBD Videos
We present an efficient and scalable algorithm for segmenting 3D RGBD point clouds by combining depth, color, and temporal information using a multistage, hierarchical graph-based approach.
Let's Dance: Learning From Online Dance Videos
We present a comparison of numerous state-of-the-art techniques on our dataset using three different representations (video, optical flow and multi-person pose data) in order to analyze these approaches.
DiscrimNet: Semi-Supervised Action Recognition from Videos using Generative Adversarial Networks
We propose an action recognition framework using Gen- erative Adversarial Networks.
Automated Surgical Skill Assessment in RMIS Training
In this paper, we explore the usage of different holistic features for automated skill assessment using only robot kinematic data and propose a weighted feature fusion technique for improving score prediction performance.
One-Shot Learning for Semantic Segmentation
Low-shot learning methods for image classification support learning from sparse data.
Semantic Instance Labeling Leveraging Hierarchical Segmentation
Most of the approaches for indoor RGBD semantic la- beling focus on using pixels or superpixels to train a classi- fier.
Eyemotion: Classifying facial expressions in VR using eye-tracking cameras
One of the main challenges of social interaction in virtual reality settings is that head-mounted displays occlude a large portion of the face, blocking facial expressions and thereby restricting social engagement cues among users.
Video and Accelerometer-Based Motion Analysis for Automated Surgical Skills Assessment
Methods: We conduct the largest study, to the best of our knowledge, for basic surgical skills assessment on a dataset that contained video and accelerometer data for suturing and knot-tying tasks.
Complex Event Recognition from Images with Few Training Examples
We propose to leverage concept-level representations for complex event recognition in photographs given limited training examples.
Discovering Picturesque Highlights from Egocentric Vacation Videos
We present an approach for identifying picturesque highlights from large amounts of egocentric video data.
Depth Extraction from Videos Using Geometric Context and Occlusion Boundaries
We propose to learn and infer depth in videos from appearance, motion, occlusion boundaries, and geometric context of the scene.
Geometric Context from Videos
We present a novel algorithm for estimating the broad 3D geometric structure of outdoor video scenes.
Finding Temporally Consistent Occlusion Boundaries in Videos using Geometric Context
Our proposed framework provides an efficient approach for finding temporally consistent occlusion boundaries in video by utilizing causality, redundancy in videos, and semantic layout of the scene.
Leveraging Context to Support Automated Food Recognition in Restaurants
The pervasiveness of mobile cameras has resulted in a dramatic increase in food photos, which are pictures reflecting what people eat.
Egocentric Field-of-View Localization Using First-Person Point-of-View Devices
We present a technique that uses images, videos and sensor data taken from first-person point-of-view devices to perform egocentric field-of-view (FOV) localization.
Augmenting Bag-of-Words: Data-Driven Discovery of Temporal and Structural Information for Activity Recognition
We present data-driven techniques to augment Bag of Words (BoW) models, which allow for more robust modeling and recognition of complex long-term activities, especially when the structure and topology of the activities are not known a priori.
Predicting Daily Activities From Egocentric Images Using Deep Learning
We collected a dataset of 40, 103 egocentric images over a 6 month period with 19 activity classes and demonstrate the benefit of state-of-the-art deep learning techniques for learning and predicting daily activities.
Beyond Sentiment: The Manifold of Human Emotions
Sentiment analysis predicts the presence of positive or negative emotions in a text document.
