Search Results for author:
Found 62 papers, 26 papers with code
AssembleNet++: Assembling Modality Representations via Attention Connections - Supplementary Material -
We create a family of powerful video models which are able to: (i) learn interactions between semantic object information and raw appearance and motion features, and (ii) deploy attention in order to better learn the importance of features at each convolutional block of the network.
Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioning
Instead, we propose an efficient, online approach which outputs frequent, detailed and temporally aligned captions, without access to future frames.
3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation
3D panoptic segmentation is a challenging perception task, especially in autonomous driving.
On Scaling Up a Multilingual Vision and Language Model
We explore the boundaries of scaling up a multilingual vision and language model both in terms of size of the components and the breadth of its training task mixture.
Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities
We propose a multimodal model, called Mirasol3B, consisting of an autoregressive component for the time-synchronized modalities (audio and video), and an autoregressive component for the context modalities which are not necessarily aligned in time but are still sequential.
Ranked #1 on
Audio Classification
on VGGSound
Region-centric Image-Language Pretraining for Open-Vocabulary Detection
We present a new open-vocabulary detection approach based on region-centric image-language pretraining to bridge the gap between image-level pretraining and open-vocabulary object detection.
Ranked #2 on
Open Vocabulary Object Detection
on LVIS v1.0
Contrastive Feature Masking Open-Vocabulary Vision Transformer
We present Contrastive Feature Masking Vision Transformer (CFM-ViT) - an image-text pretraining methodology that achieves simultaneous learning of image- and region-level representation for open-vocabulary object detection (OVD).
Ranked #8 on
Open Vocabulary Object Detection
on LVIS v1.0
Diversifying Joint Vision-Language Tokenization Learning
Building joint representations across images and text is an essential step for tasks such as Visual Question Answering and Video Question Answering.
Joint Adaptive Representations for Image-Language Learning
We here propose a much easier recipe for image-language learning, which produces effective models, outperforming bigger and more expensive ones, often trained on orders of magnitude larger datasets.
PaLI-X: On Scaling up a Multilingual Vision and Language Model
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
Ranked #1 on
Fine-Grained Image Recognition
on OVEN
Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers
We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a contrastive image-text pretraining recipe to bridge the gap between image-level pretraining and open-vocabulary object detection.
Ranked #6 on
Zero-Shot Cross-Modal Retrieval
on Flickr30k
MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks
We propose a novel paradigm of training with a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks.
Ranked #1 on
Video Captioning
on MSVD
Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning
We present a simple approach which can turn a ViT encoder into an efficient video model, which can seamlessly work with both image and video inputs.
Ranked #2 on
Action Classification
on Kinetics-600
(using extra training data)
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
We present F-VLM, a simple open-vocabulary object detection method built upon Frozen Vision and Language Models.
PaLI: A Jointly-Scaled Multilingual Language-Image Model
PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages.
Pre-training image-language transformers for open-vocabulary tasks
We present a pre-training approach for vision and language transformer models, which is based on a mixture of diverse tasks.
Video Question Answering with Iterative Video-Text Co-Tokenization
Video question answering is a challenging task that requires understanding jointly the language input, the visual information in individual video frames, as well as the temporal information about the events occurring in the video.
Ranked #4 on
Video Question Answering
on iVQA
Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering
We present Answer-Me, a task-aware multi-task framework which unifies a variety of question answering tasks, such as, visual question answering, visual entailment, visual reasoning.
FindIt: Generalized Localization with Natural Language Queries
We propose FindIt, a simple and versatile framework that unifies a variety of visual grounding and localization tasks including referring expression comprehension, text-based localization, and object detection.
Mechanical Search on Shelves using a Novel "Bluction" Tool
Shelves are common in homes, warehouses, and commercial settings due to their storage efficiency.
TokenLearner: Adaptive Space-Time Tokenization for Videos
In this paper, we introduce a novel visual representation learning which relies on a handful of adaptively learned tokens, and which is applicable to both image and video understanding tasks.
4D-Net for Learned Multi-Modal Alignment
We present 4D-Net, a 3D object detection approach, which utilizes 3D Point Cloud and RGB sensing information, both in time.
Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image
3D perception of object shapes from RGB image input is fundamental towards semantic scene understanding, grounding image-based perception in our spatially 3-dimensional real-world environments.
Learning Open-World Object Proposals without Learning to Classify
In this paper, we identify that the problem is that the binary classifiers in existing proposal methods tend to overfit to the training categories.
Ranked #2 on
Open World Object Detection
on COCO VOC to non-VOC
Unsupervised Discovery of Actions in Instructional Videos
In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos.
TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
In this paper, we introduce a novel visual representation learning which relies on a handful of adaptively learned tokens, and which is applicable to both image and video understanding tasks.
Ranked #1 on
Action Classification
on Charades
Unsupervised Action Segmentation for Instructional Videos
In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos, which are rarely annotated with atomic actions.
SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping
We present SMURF, a method for unsupervised learning of optical flow that improves state of the art on all benchmarks by $36\%$ to $40\%$ (over the prior best method UFlow) and even outperforms several supervised approaches such as PWC-Net and FlowNet2.
Adaptive Intermediate Representations for Video Understanding
A common strategy to video understanding is to incorporate spatial and motion information by fusing features derived from RGB frames and optical flow.
Ranked #5 on
Action Classification
on Toyota Smarthome dataset
Visionary: Vision architecture discovery for robot learning
We propose a vision-based architecture search algorithm for robot manipulation learning, which discovers interactions between low dimension action inputs and high dimensional visual inputs.
Unsupervised Monocular Depth Learning in Dynamic Scenes
We present a method for jointly training the estimation of depth, ego-motion, and a dense 3D translation field of objects relative to the scene, with monocular photometric consistency being the sole source of supervision.
Ranked #10 on
Unsupervised Monocular Depth Estimation
on Cityscapes
AssembleNet++: Assembling Modality Representations via Attention Connections
We create a family of powerful video models which are able to: (i) learn interactions between semantic object information and raw appearance and motion features, and (ii) deploy attention in order to better learn the importance of features at each convolutional block of the network.
Ranked #4 on
Action Classification
on Toyota Smarthome dataset
Adversarial Generative Grammars for Human Activity Prediction
In this paper we propose an adversarial generative grammar model for future prediction.
Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve
We propose to leverage existing large-scale datasets of 3D models to understand the underlying 3D structure of objects seen in an image by constructing a CAD-based representation of the objects and their poses.
AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification
The discovered attention cells can be seamlessly inserted into existing backbone networks, e. g., I3D or S3D, and improve video classification accuracy by more than 2% on both Kinetics-600 and MiT datasets.
What Matters in Unsupervised Optical Flow
We systematically compare and analyze a set of key components in unsupervised optical flow to identify which photometric loss, occlusion handling, and smoothness regularization is most effective.
Ranked #5 on
Optical Flow Estimation
on Sintel Clean unsupervised
Differentiable Mapping Networks: Learning Structured Map Representations for Sparse Visual Localization
We address these tasks by combining spatial structure (differentiable mapping) and end-to-end learning in a novel neural network architecture: the Differentiable Mapping Network (DMN).
Taskology: Utilizing Task Relations at Scale
Many computer vision tasks address the problem of scene understanding and are naturally interrelated e. g. object classification, detection, scene segmentation, depth estimation, etc.
X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions
For applications in e-commerce, warehouses, healthcare, and home service, robots are often required to search through heaps of objects to grasp a specific target object.
Improving Semantic Segmentation through Spatio-Temporal Consistency Learned from Videos
We leverage unsupervised learning of depth, egomotion, and camera intrinsics to improve the performance of single-image semantic segmentation, by enforcing 3D-geometric and temporal consistency of segmentation masks across video frames.
Evolving Losses for Unsupervised Video Representation Learning
We present a new method to learn video representations from large-scale unlabeled video data.
SPIN: A High Speed, High Resolution Vision Dataset for Tracking and Action Recognition in Ping Pong
The corpus consists of ping pong play with three main annotation streams that can be used to learn tracking and action recognition models -- tracking of the ping pong ball and poses of humans in the videos and the spin of the ball being hit by humans.
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects
We address two problems: first, we establish an easy method for capturing and labeling 3D keypoints on desktop objects with an RGB camera; and second, we develop a deep neural network, called $KeyPose$, that learns to accurately predict object poses using 3D keypoints, from stereo input, and works even for transparent objects.
Tiny Video Networks
Video understanding is a challenging problem with great impact on the abilities of autonomous agents working in the real-world.
Unsupervised Monocular Depth and Ego-motion Learning with Structure and Semantics
We present an approach which takes advantage of both structure and semantics for unsupervised monocular learning of depth and ego-motion.
Evolving Losses for Unlabeled Video Representation Learning
We present a new method to learn video representations from unlabeled data.
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
Learning to represent videos is a very challenging task both algorithmically and computationally.
Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras
We present a novel method for simultaneous learning of depth, egomotion, object motion, and camera intrinsics from monocular videos, using only consistency across neighboring video frames as supervision signal.
Ranked #11 on
Unsupervised Monocular Depth Estimation
on Cityscapes
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors
However, it is difficult and costly to segment objects in novel categories because a large number of mask annotations is required.
Differentiable Grammars for Videos
This paper proposes a novel algorithm which learns a formal regular grammar from real-world continuous data, such as videos.
Future Segmentation Using 3D Structure
Predicting the future to anticipate the outcome of events and actions is a critical attribute of autonomous agents; particularly for agents which must rely heavily on real time visual data for decision making.
Probabilistic Object Detection: Definition and Evaluation
We introduce Probabilistic Object Detection, the task of detecting objects in images and accurately quantifying the spatial and semantic uncertainties of the detections.
Evolving Space-Time Neural Architectures for Videos
We present a new method for finding video CNN architectures that capture rich spatio-temporal information in videos.
Ranked #20 on
Action Classification
on MiT
Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos
Models and examples built with TensorFlow
Ranked #12 on
Unsupervised Monocular Depth Estimation
on Cityscapes
Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints
We present a novel approach for unsupervised learning of depth and ego-motion from monocular video.
Object category learning and retrieval with weak supervision
We consider the problem of retrieving objects from image data and learning to classify them into meaningful semantic categories with minimal supervision.
Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs
We approach structured output prediction by optimizing a deep value network (DVN) to precisely estimate the task loss on different output configurations for a given input.
Improved generator objectives for GANs
We present a framework to understand GAN training as alternating density ratio estimation and approximate divergence minimization.
Geometry-Based Next Frame Prediction from Monocular Video
We consider the problem of next frame prediction from video input.
Object Recognition from Short Videos for Robotic Perception
Videos, unlike still images, are temporally coherent which makes the application of deep networks non-trivial.
Real-Time Grasp Detection Using Convolutional Neural Networks
We present an accurate, real-time approach to robotic grasp detection based on convolutional neural networks.
Ranked #5 on
Robotic Grasping
on Cornell Grasp Dataset
Efficient Object Detection and Segmentation for Fine-Grained Recognition
The algorithm first detects low-level regions that could potentially belong to the object and then performs a full-object segmentation through propagation.
