Search Results for author:
Found 60 papers, 25 papers with code
Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints
We present a novel approach for unsupervised learning of depth and ego-motion from monocular video.
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
Learning to represent videos is a very challenging task both algorithmically and computationally.
Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos
Models and examples built with TensorFlow
Ranked #10 on
Unsupervised Monocular Depth Estimation
on Cityscapes
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
We present F-VLM, a simple open-vocabulary object detection method built upon Frozen Vision and Language Models.
Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers
We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a contrastive image-text pretraining recipe to bridge the gap between image-level pretraining and open-vocabulary object detection.
Ranked #5 on
Zero-Shot Cross-Modal Retrieval
on Flickr30k
Detection-Oriented Image-Text Pretraining for Open-Vocabulary Detection
We present a new open-vocabulary detection approach based on detection-oriented image-text pretraining to bridge the gap between image-level pretraining and open-vocabulary object detection.
Ranked #1 on
Open Vocabulary Object Detection
on LVIS v1.0
Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras
We present a novel method for simultaneous learning of depth, egomotion, object motion, and camera intrinsics from monocular videos, using only consistency across neighboring video frames as supervision signal.
Unsupervised Monocular Depth Learning in Dynamic Scenes
We present a method for jointly training the estimation of depth, ego-motion, and a dense 3D translation field of objects relative to the scene, with monocular photometric consistency being the sole source of supervision.
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects
We address two problems: first, we establish an easy method for capturing and labeling 3D keypoints on desktop objects with an RGB camera; and second, we develop a deep neural network, called $KeyPose$, that learns to accurately predict object poses using 3D keypoints, from stereo input, and works even for transparent objects.
What Matters in Unsupervised Optical Flow
We systematically compare and analyze a set of key components in unsupervised optical flow to identify which photometric loss, occlusion handling, and smoothness regularization is most effective.
Ranked #5 on
Optical Flow Estimation
on Sintel Clean unsupervised
AssembleNet++: Assembling Modality Representations via Attention Connections
We create a family of powerful video models which are able to: (i) learn interactions between semantic object information and raw appearance and motion features, and (ii) deploy attention in order to better learn the importance of features at each convolutional block of the network.
Ranked #4 on
Action Classification
on Toyota Smarthome dataset
Tiny Video Networks
Video understanding is a challenging problem with great impact on the abilities of autonomous agents working in the real-world.
SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping
We present SMURF, a method for unsupervised learning of optical flow that improves state of the art on all benchmarks by $36\%$ to $40\%$ (over the prior best method UFlow) and even outperforms several supervised approaches such as PWC-Net and FlowNet2.
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors
However, it is difficult and costly to segment objects in novel categories because a large number of mask annotations is required.
TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
In this paper, we introduce a novel visual representation learning which relies on a handful of adaptively learned tokens, and which is applicable to both image and video understanding tasks.
Ranked #1 on
Action Classification
on Charades
PaLI: A Jointly-Scaled Multilingual Language-Image Model
PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages.
Learning Open-World Object Proposals without Learning to Classify
In this paper, we identify that the problem is that the binary classifiers in existing proposal methods tend to overfit to the training categories.
Ranked #2 on
Open World Object Detection
on COCO VOC to non-VOC
Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning
We present a simple approach which can turn a ViT encoder into an efficient video model, which can seamlessly work with both image and video inputs.
Ranked #2 on
Action Classification
on Kinetics-600
(using extra training data)
PaLI-X: On Scaling up a Multilingual Vision and Language Model
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
Ranked #1 on
Fine-Grained Image Recognition
on OVEN
TokenLearner: Adaptive Space-Time Tokenization for Videos
In this paper, we introduce a novel visual representation learning which relies on a handful of adaptively learned tokens, and which is applicable to both image and video understanding tasks.
Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs
We approach structured output prediction by optimizing a deep value network (DVN) to precisely estimate the task loss on different output configurations for a given input.
4D-Net for Learned Multi-Modal Alignment
We present 4D-Net, a 3D object detection approach, which utilizes 3D Point Cloud and RGB sensing information, both in time.
Real-Time Grasp Detection Using Convolutional Neural Networks
We present an accurate, real-time approach to robotic grasp detection based on convolutional neural networks.
Ranked #5 on
Robotic Grasping
on Cornell Grasp Dataset
Probabilistic Object Detection: Definition and Evaluation
We introduce Probabilistic Object Detection, the task of detecting objects in images and accurately quantifying the spatial and semantic uncertainties of the detections.
Object category learning and retrieval with weak supervision
We consider the problem of retrieving objects from image data and learning to classify them into meaningful semantic categories with minimal supervision.
Geometry-Based Next Frame Prediction from Monocular Video
We consider the problem of next frame prediction from video input.
Improved generator objectives for GANs
We present a framework to understand GAN training as alternating density ratio estimation and approximate divergence minimization.
Object Recognition from Short Videos for Robotic Perception
Videos, unlike still images, are temporally coherent which makes the application of deep networks non-trivial.
Evolving Space-Time Neural Architectures for Videos
We present a new method for finding video CNN architectures that capture rich spatio-temporal information in videos.
Ranked #22 on
Action Classification
on MiT
Future Segmentation Using 3D Structure
Predicting the future to anticipate the outcome of events and actions is a critical attribute of autonomous agents; particularly for agents which must rely heavily on real time visual data for decision making.
Efficient Object Detection and Segmentation for Fine-Grained Recognition
The algorithm first detects low-level regions that could potentially belong to the object and then performs a full-object segmentation through propagation.
Differentiable Grammars for Videos
This paper proposes a novel algorithm which learns a formal regular grammar from real-world continuous data, such as videos.
Evolving Losses for Unlabeled Video Representation Learning
We present a new method to learn video representations from unlabeled data.
Unsupervised Monocular Depth and Ego-motion Learning with Structure and Semantics
We present an approach which takes advantage of both structure and semantics for unsupervised monocular learning of depth and ego-motion.
SPIN: A High Speed, High Resolution Vision Dataset for Tracking and Action Recognition in Ping Pong
The corpus consists of ping pong play with three main annotation streams that can be used to learn tracking and action recognition models -- tracking of the ping pong ball and poses of humans in the videos and the spin of the ball being hit by humans.
Evolving Losses for Unsupervised Video Representation Learning
We present a new method to learn video representations from large-scale unlabeled video data.
Improving Semantic Segmentation through Spatio-Temporal Consistency Learned from Videos
We leverage unsupervised learning of depth, egomotion, and camera intrinsics to improve the performance of single-image semantic segmentation, by enforcing 3D-geometric and temporal consistency of segmentation masks across video frames.
X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions
For applications in e-commerce, warehouses, healthcare, and home service, robots are often required to search through heaps of objects to grasp a specific target object.
Taskology: Utilizing Task Relations at Scale
Many computer vision tasks address the problem of scene understanding and are naturally interrelated e. g. object classification, detection, scene segmentation, depth estimation, etc.
Differentiable Mapping Networks: Learning Structured Map Representations for Sparse Visual Localization
We address these tasks by combining spatial structure (differentiable mapping) and end-to-end learning in a novel neural network architecture: the Differentiable Mapping Network (DMN).
AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification
The discovered attention cells can be seamlessly inserted into existing backbone networks, e. g., I3D or S3D, and improve video classification accuracy by more than 2% on both Kinetics-600 and MiT datasets.
Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve
We propose to leverage existing large-scale datasets of 3D models to understand the underlying 3D structure of objects seen in an image by constructing a CAD-based representation of the objects and their poses.
Adversarial Generative Grammars for Human Activity Prediction
In this paper we propose an adversarial generative grammar model for future prediction.
AssembleNet++: Assembling Modality Representations via Attention Connections - Supplementary Material -
We create a family of powerful video models which are able to: (i) learn interactions between semantic object information and raw appearance and motion features, and (ii) deploy attention in order to better learn the importance of features at each convolutional block of the network.
Visionary: Vision architecture discovery for robot learning
We propose a vision-based architecture search algorithm for robot manipulation learning, which discovers interactions between low dimension action inputs and high dimensional visual inputs.
Adaptive Intermediate Representations for Video Understanding
A common strategy to video understanding is to incorporate spatial and motion information by fusing features derived from RGB frames and optical flow.
Ranked #5 on
Action Classification
on Toyota Smarthome dataset
Unsupervised Action Segmentation for Instructional Videos
In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos, which are rarely annotated with atomic actions.
Unsupervised Discovery of Actions in Instructional Videos
In this paper we address the problem of automatically discovering atomic actions in unsupervised manner from instructional videos.
Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image
3D perception of object shapes from RGB image input is fundamental towards semantic scene understanding, grounding image-based perception in our spatially 3-dimensional real-world environments.
Mechanical Search on Shelves using a Novel "Bluction" Tool
Shelves are common in homes, warehouses, and commercial settings due to their storage efficiency.
FindIt: Generalized Localization with Natural Language Queries
We propose FindIt, a simple and versatile framework that unifies a variety of visual grounding and localization tasks including referring expression comprehension, text-based localization, and object detection.
Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering
We present Answer-Me, a task-aware multi-task framework which unifies a variety of question answering tasks, such as, visual question answering, visual entailment, visual reasoning.
Video Question Answering with Iterative Video-Text Co-Tokenization
Video question answering is a challenging task that requires understanding jointly the language input, the visual information in individual video frames, as well as the temporal information about the events occurring in the video.
Ranked #4 on
Video Question Answering
on iVQA
Pre-training image-language transformers for open-vocabulary tasks
We present a pre-training approach for vision and language transformer models, which is based on a mixture of diverse tasks.
MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks
We propose a novel paradigm of training with a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks.
Joint Adaptive Representations for Image-Language Learning
We here propose a much easier recipe for image-language learning, which produces effective models, outperforming bigger and more expensive ones, often trained on orders of magnitude larger datasets.
Diversifying Joint Vision-Language Tokenization Learning
Building joint representations across images and text is an essential step for tasks such as Visual Question Answering and Video Question Answering.
Contrastive Feature Masking Open-Vocabulary Vision Transformer
We present Contrastive Feature Masking Vision Transformer (CFM-ViT) - an image-text pretraining methodology that achieves simultaneous learning of image- and region-level representation for open-vocabulary object detection (OVD).
Ranked #5 on
Open Vocabulary Object Detection
on LVIS v1.0
Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities
We propose a multimodal model, called Mirasol3B, consisting of an autoregressive component for the time-synchronized modalities (audio and video), and an autoregressive component for the context modalities which are not necessarily aligned in time but are still sequential.
Ranked #1 on
Audio Classification
on VGGSound
3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation
3D panoptic segmentation is a challenging perception task, especially in autonomous driving.
