Search Results for author:
Found 57 papers, 22 papers with code
AssembleNet++: Assembling Modality Representations via Attention Connections - Supplementary Material -
We create a family of powerful video models which are able to: (i) learn interactions between semantic object information and raw appearance and motion features, and (ii) deploy attention in order to better learn the importance of features at each convolutional block of the network.
Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors
Recent diffusion-based generative models combined with vision-language models are capable of creating realistic images from natural language prompts.
Token Turing Machines
We propose Token Turing Machines (TTM), a sequential, autoregressive Transformer model with memory for real-world sequential visual understanding.
Ranked #1 on
Action Detection
on Charades
Grafting Vision Transformers
In this paper, we present a simple and efficient add-on component (termed GrafT) that considers global dependencies and multi-scale information throughout the network, in both high- and low-resolution features alike.
Open-vocabulary Queryable Scene Representations for Real World Planning
Large language models (LLMs) have unlocked new capabilities of task planning from human instructions.
Video Question Answering with Iterative Video-Text Co-Tokenization
Video question answering is a challenging task that requires understanding jointly the language input, the visual information in individual video frames, as well as the temporal information about the events occurring in the video.
Ranked #4 on
Video Question Answering
on iVQA
Video + CLIP Baseline for Ego4D Long-term Action Anticipation
The CLIP embedding provides fine-grained understanding of objects relevant for an action whereas the slowfast network is responsible for modeling temporal information within a video clip of few frames.
Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space
To this end, we propose a 3D Token Representation Layer (3DTRL) that estimates the 3D positional information of the visual tokens and leverages it for learning viewpoint-agnostic representations.
Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?
We investigate whether self-supervised learning (SSL) can improve online reinforcement learning (RL) from pixels.
ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints
Learning self-supervised video representation predominantly focuses on discriminating instances generated from simple data augmentation schemes.
STC-mix: Space, Time, Channel mixing for Self-supervised Video Representation
To this end, we propose Cross-Modal Manifold Cutmix (CMMC) that inserts a video tesseract into another video tesseract in the feature space across two different modalities.
MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection
Action detection is an essential and challenging task, especially for densely labelled datasets of untrimmed videos.
Ranked #2 on
Action Detection
on TSU
SWAT: Spatial Structure Within and Among Tokens
Modeling visual data as tokens (i. e., image patches) using attention mechanisms, feed-forward networks or convolutions has been highly effective in recent years.
Weakly-guided Self-supervised Pretraining for Temporal Activity Detection
However, such pretrained models are not ideal for downstream detection, due to the disparity between the pretraining and the downstream fine-tuning tasks.
Ranked #3 on
Action Detection
on Charades
StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning
Reinforcement Learning (RL) can be considered as a sequence modeling task: given a sequence of past state-action-reward experiences, an agent predicts a sequence of next actions.
4D-Net for Learned Multi-Modal Alignment
We present 4D-Net, a 3D object detection approach, which utilizes 3D Point Cloud and RGB sensing information, both in time.
Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learning
Third-person imitation learning (TPIL) is the concept of learning action policies by observing other agents in a third-person view (TPV), similar to what humans do.
Unsupervised Discovery of Actions in Instructional Videos
In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos.
TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
In this paper, we introduce a novel visual representation learning which relies on a handful of adaptively learned tokens, and which is applicable to both image and video understanding tasks.
Ranked #1 on
Action Classification
on Charades
Unsupervised Action Segmentation for Instructional Videos
In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos, which are rarely annotated with atomic actions.
Recognizing Actions in Videos from Unseen Viewpoints
Standard methods for video recognition use large CNNs designed to capture spatio-temporal data.
Ranked #2 on
Action Classification
on Toyota Smarthome dataset
(CV1 metric)
Visionary: Vision architecture discovery for robot learning
We propose a vision-based architecture search algorithm for robot manipulation learning, which discovers interactions between low dimension action inputs and high dimensional visual inputs.
Coarse-Fine Networks for Temporal Activity Detection in Videos
In this paper, we introduce Coarse-Fine Networks, a two-stream architecture which benefits from different abstractions of temporal resolution to learn better video representations for long-term motion.
Ranked #7 on
Action Detection
on Charades
Reducing Inference Latency with Concurrent Architectures for Image Recognition
Satisfying the high computation demand of modern deep learning architectures is challenging for achieving low inference latency.
AssembleNet++: Assembling Modality Representations via Attention Connections
We create a family of powerful video models which are able to: (i) learn interactions between semantic object information and raw appearance and motion features, and (ii) deploy attention in order to better learn the importance of features at each convolutional block of the network.
Ranked #4 on
Action Classification
on Toyota Smarthome dataset
Adversarial Generative Grammars for Human Activity Prediction
In this paper we propose an adversarial generative grammar model for future prediction.
AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification
The discovered attention cells can be seamlessly inserted into existing backbone networks, e. g., I3D or S3D, and improve video classification accuracy by more than 2% on both Kinetics-600 and MiT datasets.
AViD Dataset: Anonymized Videos from Diverse Countries
We confirm that most of the existing video datasets are statistically biased to only capture action videos from a limited number of countries.
Ranked #2 on
Action Classification
on AViD
LCP: A Low-Communication Parallelization Method for Fast Neural Network Inference in Image Recognition
To benefit from available compute resources with low communication overhead, we propose the first DNN parallelization method for reducing the communication overhead in a distributed system.
Evolving Losses for Unsupervised Video Representation Learning
We present a new method to learn video representations from large-scale unlabeled video data.
Password-conditioned Anonymization and Deanonymization with Face Identity Transformers
Cameras are prevalent in our daily lives, and enable many useful systems built upon computer vision technologies such as smart cameras and home robots for service applications.
Tiny Video Networks
Video understanding is a challenging problem with great impact on the abilities of autonomous agents working in the real-world.
Model-based Behavioral Cloning with Future Image Similarity Learning
We present a visual imitation learning framework that enables learning of robot action policies solely based on expert samples without any robot trials.
Evolving Losses for Unlabeled Video Representation Learning
We present a new method to learn video representations from unlabeled data.
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
Learning to represent videos is a very challenging task both algorithmically and computationally.
Unseen Action Recognition with Unpaired Adversarial Multimodal Learning
In this paper, we present a method to learn a joint multimodal representation space that allows for the recognition of unseen activities in videos.
Differentiable Grammars for Videos
This paper proposes a novel algorithm which learns a formal regular grammar from real-world continuous data, such as videos.
Evolving Space-Time Neural Architectures for Videos
We present a new method for finding video CNN architectures that capture rich spatio-temporal information in videos.
Ranked #20 on
Action Classification
on Moments in Time
Representation Flow for Action Recognition
Our representation flow layer is a fully-differentiable layer designed to capture the `flow' of any representation channel within a convolutional neural network for action recognition.
Ranked #14 on
Action Recognition
on HMDB-51
Learning Multimodal Representations for Unseen Activities
We present a method to learn a joint multimodal representation space that enables recognition of unseen activities in videos.
Learning Real-World Robot Policies by Dreaming
Learning to control robots directly based on images is a primary challenge in robotics.
Fine-grained Activity Recognition in Baseball Videos
In this paper, we introduce a challenging new dataset, MLB-YouTube, designed for fine-grained activity detection.
Learning to Anonymize Faces for Privacy Preserving Action Detection
The end result is a video anonymizer that performs pixel-level modifications to anonymize each person's face, with minimal effect on action detection performance.
Joint Person Segmentation and Identification in Synchronized First- and Third-person Videos
In this paper, we wish to solve two specific problems: (1) given two or more synchronized third-person videos of a scene, produce a pixel-level segmentation of each visible person and identify corresponding people across different views (i. e., determine who in camera A corresponds with whom in camera B), and (2) given one or more synchronized third-person videos as well as a first-person video taken by a mobile or wearable camera, segment and identify the camera wearer in the third-person videos.
Temporal Gaussian Mixture Layer for Videos
We introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer and present how it can be used to efficiently capture longer-term temporal information in continuous activity videos.
Ranked #4 on
Action Detection
on Multi-THUMOS
Musical Chair: Efficient Real-Time Recognition Using Collaborative IoT Devices
Furthermore, in image recognition, Musical Chair achieves similar performance and saves dynamic energy.
Learning Latent Super-Events to Detect Multiple Activities in Videos
In this paper, we introduce the concept of learning latent super-events from activity videos, and present how it benefits activity detection in continuous videos.
Ranked #6 on
Action Detection
on Multi-THUMOS
Extreme Low Resolution Activity Recognition with Multi-Siamese Embedding Learning
This paper presents an approach for recognizing human activities from extreme low resolution (e. g., 16x12) videos.
Forecasting Hands and Objects in Future Frames
The key idea is that (1) an intermediate representation of a convolutional object recognition model abstracts scene information in its frame and that (2) we can predict (i. e., regress) such representations corresponding to the future frames based on that of the current frame.
Identifying First-person Camera Wearers in Third-person Videos
We consider scenarios in which we wish to perform joint scene understanding, object tracking, activity recognition, and other tasks in environments in which multiple people are wearing body-worn cameras while a third-person static camera also captures the scene.
Learning Robot Activities from First-Person Human Videos Using Convolutional Future Regression
We design a new approach that allows robot learning of new activities from unlabeled human example videos.
Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions
In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer the grammar to humanoids to enable a real-time motion inference for human-robot interaction (HRI).
Learning Latent Sub-events in Activity Videos Using Temporal Attention Filters
In this paper, we newly introduce the concept of temporal attention filters, and describe how they can be used for human activity recognition from videos.
Ranked #1 on
Activity Recognition In Videos
on DogCentric
Privacy-Preserving Human Activity Recognition from Extreme Low Resolution
We introduce the paradigm of inverse super resolution (ISR), the concept of learning the optimal set of image transformations to generate multiple low-resolution (LR) training videos from a single video.
Multi-Type Activity Recognition in Robot-Centric Scenarios
Activity recognition is very useful in scenarios where robots interact with, monitor or assist humans.
First-Person Activity Recognition: What Are They Doing to Me?
This paper discusses the problem of recognizing interaction-level human activities from a first-person viewpoint.
