Search Results for author:
Found 21 papers, 12 papers with code
TAPNext: Tracking Any Point (TAP) as Next Token Prediction
Tracking Any Point (TAP) in a video is a challenging computer vision problem with many demonstrated applications in robotics, video editing, and 3D reconstruction.
Scaling 4D Representations
Scaling has not yet been convincingly demonstrated for pure self-supervised learning from video.
PaliGemma: A versatile 3B VLM for transfer
PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model.
TAPVid-3D: A Benchmark for Tracking Any Point in 3D
We introduce a new benchmark, TAPVid-3D, for evaluating the task of long-range Tracking Any Point in 3D (TAP-3D).
Memory Consolidation Enables Long-Context Video Understanding
Most transformer-based video encoders are limited to short temporal contexts due to their quadratic complexity.
BootsTAP: Bootstrapped Training for Tracking-Any-Point
To endow models with greater understanding of physics and motion, it is useful to enable them to perceive how solid surfaces move and deform in real scenes.
Ranked #1 on
Point Tracking
on TAP-Vid-RGB-Stacking
A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames
Understanding long, real-world videos requires modeling of long-range visual dependencies.
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e. g. Flamingo, SeViLA, or GPT-4).
Ranked #1 on
Point Tracking
on Perception Test
Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation
We demonstrate that appropriate placement of our parameter efficient adapters can significantly reduce the performance gap between frozen pretrained representations and full end-to-end fine-tuning without changes to the original representation and thus preserving original capabilities of the pretrained model.
Perception Test: A Diagnostic Benchmark for Multimodal Models
We propose a novel multimodal benchmark – the Perception Test – that aims to extensively evaluate perception and reasoning skills of multimodal models.
Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methods
Self-supervised methods have achieved remarkable success in transfer learning, often achieving the same or better accuracy than supervised pre-training.
Object discovery and representation networks
The promise of self-supervised learning (SSL) is to leverage large amounts of unlabeled data to solve complex tasks.
HiP: Hierarchical Perceiver
This however hinders them from scaling up to the inputs sizes required to process raw high-resolution images or video.
EcoFlow: Efficient Convolutional Dataflows for Low-Power Neural Network Accelerators
We find that commonly-used low-power CNN inference accelerators based on spatial architectures are not optimized for both of these convolutional kernels.
Perceiver IO: A General Architecture for Structured Inputs & Outputs
A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible.
Ranked #1 on
Optical Flow Estimation
on KITTI 2015
(Average End-Point Error metric)
Efficient Visual Pretraining with Contrastive Detection
Self-supervised pretraining has been shown to yield powerful representations for transfer learning.
Ranked #65 on
Semantic Segmentation
on Cityscapes val
(using extra training data)
A Deep Learning Approach for Characterizing Major Galaxy Mergers
Fine-grained estimation of galaxy merger stages from observations is a key problem useful for validation of our current theoretical understanding of galaxy formation.
Accurate, Low-Latency Visual Perception for Autonomous Racing:Challenges, Mechanisms, and Practical Solutions
Autonomous racing provides the opportunity to test safety-critical perception pipelines at their limit.
EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference Using Approximate DRAM
Based on this observation, we propose EDEN, a general framework that reduces DNN energy consumption and DNN evaluation latency by using approximate DRAM devices, while strictly meeting a user-specified target DNN accuracy.
Understanding Recurrent Neural State Using Memory Signatures
We demonstrate this method's usefulness in revealing information divergence in the bases of recurrent factorized kernels, visualizing the character-level differences between the memory of n-gram and recurrent language models, and extracting knowledge of history encoded in the layers of grapheme-based end-to-end ASR networks.
Learning a CNN-based End-to-End Controller for a Formula SAE Racecar
We present a set of CNN-based end-to-end models for controls of a Formula SAE racecar, along with various benchmarking and visualization tools to understand model performance.
