Search Results for author:
Found 116 papers, 54 papers with code
Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting
One promising self-supervised task is 3D point cloud forecasting from unannotated LiDAR sequences.
3D-aware Conditional Image Synthesis
We propose pix2pix3D, a 3D-aware conditional generative model for controllable photorealistic image synthesis.
Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models
By repurposing class names as additional one-shot training samples, we achieve SOTA results with an embarrassingly simple linear classifier for vision-language adaptation.
Pix2Map: Cross-modal Retrieval for Inferring Street Maps from Images
Moreover, we show that our retrieved maps can be used to update or expand existing maps and even show proof-of-concept results for visual localization and image retrieval from spatial graphs.
TarViS: A Unified Approach for Target-based Video Segmentation
A single TarViS model can be trained jointly on a collection of datasets spanning different tasks, and can hot-swap between tasks during inference without any task-specific retraining.
Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting
Models are tasked with the prediction of future motion for "scored actors" in each scenario and are provided with track histories that capture object location, heading, velocity, and category.
Far3Det: Towards Far-Field 3D Detection
We focus on the task of far-field 3D detection (Far3Det) of objects beyond a certain distance from an observer, e. g., $>$50m.
Towards Long-Tailed 3D Detection
Moreover, semantic classes are often organized within a hierarchy, e. g., tail classes such as child and construction-worker are arguably subclasses of pedestrian.
Soft Augmentation for Image Classification
We draw inspiration from human visual classification studies and propose generalizing augmentation with invariant transforms to soft augmentation where the learning target softens non-linearly as a function of the degree of the transform applied to the sample: e. g., more aggressive image crop augmentations produce less confident learning targets.
Learning to Discover and Detect Objects
We then train our network to learn to classify each RoI, either as one of the known classes, seen in the source dataset, or one of the novel classes, with a long-tail distribution constraint on the class assignments, reflecting the natural frequency of classes in the real world.
Continual Learning with Evolving Class Ontologies
LECO requires learning classifiers in distinct time periods (TPs); each TP introduces a new ontology of "fine" labels that refines old ontologies of "coarse" labels (e. g., dog breeds that refine the previous ${\tt dog}$).
Differentiable Raycasting for Self-supervised Occupancy Forecasting
Self-supervised representations proposed for large-scale planning, such as ego-centric freespace, confound these two motions, making the representation difficult to use for downstream motion planners.
BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video
Multiple existing benchmarks involve tracking and segmenting objects in video e. g., Video Object Segmentation (VOS) and Multi-Object Tracking and Segmentation (MOTS), but there is little interaction between them due to the use of disparate benchmark datasets and metrics (e. g. J&F, mAP, sMOTSA).
Multi-Object Tracking
Multi-Object Tracking and Segmentation
+4
RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild
We describe a data-driven method for inferring the camera viewpoints given multiple images of an arbitrary object.
Differentiable Soft-Masked Attention
Recently, "Masked Attention" was proposed in which a given object representation only attends to those image pixel features for which the segmentation mask of that object is active.
Forecasting from LiDAR via Future Object Detection
Object detection and forecasting are fundamental components of embodied perception.
Long-Tailed Recognition via Weight Balancing
In contrast, weight decay penalizes larger weights more heavily and so learns small balanced weights; the MaxNorm constraint encourages growing small weights within a norm ball but caps all the weights by the radius.
Ranked #6 on
Long-tail Learning
on CIFAR-100-LT (ρ=10)
The CLEAR Benchmark: Continual LEArning on Real-World Imagery
The major strength of CLEAR over prior CL benchmarks is the smooth temporal evolution of visual concepts with real-world imagery, including both high-quality labeled data along with abundant unlabeled samples per time period for continual semi-supervised learning.
Opening Up Open World Tracking
A benchmark that would allow us to perform an apple-to-apple comparison of existing efforts is a crucial first step towards advancing this important research field.
BANMo: Building Animatable 3D Neural Models from Many Casual Videos
Our key insight is to merge three schools of thought; (1) classic deformable shape models that make use of articulated bones and blend skinning, (2) volumetric neural radiance fields (NeRFs) that are amenable to gradient-based optimization, and (3) canonical embeddings that generate correspondences between pixels and an articulated model.
Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs
We use neural radiance fields (NeRFs) to build interactive 3D environments from large-scale visual captures spanning buildings or even multiple city blocks collected primarily from drones.
HODOR: High-level Object Descriptors for Object Re-segmentation in Video Learned from Static Images
Existing state-of-the-art methods for Video Object Segmentation (VOS) learn low-level pixel-to-pixel correspondences between frames to propagate object masks across video.
ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction
The surface embeddings are implemented as coordinate-based MLPs that are fit to each video via consistency and contrastive reconstruction losses. Experimental results show that ViSER compares favorably against prior work on challenging videos of humans with loose clothing and unusual poses as well as animals videos from DAVIS and YTVOS.
3D Shape Reconstruction
3D Shape Reconstruction from Videos
+1
NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild
NeRS learns a neural shape representation of a closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions.
Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories
For machine learning models trained with limited labeled training data, validation stands to become the main bottleneck to reducing overall annotation costs.
FOVEA: Foveated Image Magnification for Autonomous Navigation
Efficient processing of high-res video streams is safety-critical for many robotics applications such as autonomous driving.
Depth-supervised NeRF: Fewer Views and Faster Training for Free
Crucially, SFM also produces sparse 3D points that can be used as "free" depth supervision during training: we add a loss to encourage the distribution of a ray's terminating depth matches a given 3D keypoint, incorporating depth uncertainty.
Safe Local Motion Planning With Self-Supervised Freespace Forecasting
Finally, we propose future freespace as an additional source of annotation-free supervision.
LASR: Learning Articulated Shape Reconstruction from a Monocular Video
Remarkable progress has been made in 3D reconstruction of rigid structures from a video or a collection of images.
Opening up Open-World Tracking
We hope to open a new front in multi-object tracking research that will hopefully bring us a step closer to intelligent systems that can operate safely in the real world.
Multimodal Object Detection via Probabilistic Ensembling
Object detection with multimodal inputs can improve many safety-critical systems such as autonomous vehicles (AVs).
Streaming Self-Training via Domain-Agnostic Unlabeled Images
We present streaming self-training (SST) that aims to democratize the process of learning visual recognition models such that a non-expert user can define a new task depending on their needs via a few labeled examples and minimal domain knowledge.
OpenGAN: Open-Set Recognition via Open Data Generation
However, the former generalizes poorly to diverse open test data due to overfitting to the training outliers, which are unlikely to exhaustively span the open-world.
Video-Specific Autoencoders for Exploring, Editing and Transmitting Videos
Associating latent codes of a video and manifold projection enables users to make desired edits.
Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details
On one hand, this is desirable as it treats all classes equally.
Learning to Segment Rigid Motions from Two Frames
Geometric motion segmentation algorithms, however, generalize to novel scenes, but have yet to achieve comparable performance to appearance-based ones, due to noisy motion estimations and degenerate motion configurations.
Learning Rare Category Classifiers on a Tight Labeling Budget
In this paper, we consider the scenario where we start with as-little-as five labeled positives of a rare category and a large amount of unlabeled data of which 99. 9% of it is negatives.
An Empirical Exploration of Open-Set Recognition via Lightweight Statistical Pipelines
Machine-learned safety-critical systems need to be self-aware and reliably know their unknowns in the open-world.
Detecting Invisible People
We demonstrate that current detection and tracking systems perform dramatically worse on this task.
Background Splitting: Finding Rare Classes in a Sea of Background
We focus on the real-world problem of training accurate deep models for image classification of a small number of rare categories.
What-If Motion Prediction for Autonomous Driving
Forecasting the long-term future motion of road actors is a core challenge to the deployment of safe autonomous vehicles (AVs).
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild
We present a method that infers spatial arrangements and shapes of humans and objects in a globally consistent 3D scene, all from a single image in-the-wild captured in an uncontrolled environment.
3D Human Pose Estimation
3D Shape Reconstruction From A Single 2D Image
+2
4D Visualization of Dynamic Events from Unconstrained Multi-View Videos
We present a data-driven approach for 4D space-time visualization of dynamic events from videos captured by hand-held multiple cameras.
Towards Streaming Perception
While past work has studied the algorithmic trade-off between latency and accuracy, there has not been a clear metric to compare different methods along the Pareto optimal latency-accuracy curve.
Ranked #2 on
Real-Time Object Detection
on Argoverse-HD (Detection-Only, Val)
(using extra training data)
TAO: A Large-Scale Benchmark for Tracking Any Object
To this end, we ask annotators to label objects that move at any point in the video, and give names to them post factum.
CATER: A diagnostic dataset for Compositional Actions & TEmporal Reasoning
In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved.
Learning Generative Models of Tissue Organization with Supervised GANs
A key step in understanding the spatial organization of cells and tissues is the ability to construct generative models that accurately reflect that organization.
Unsupervised Audiovisual Synthesis via Exemplar Autoencoders
We present an unsupervised approach that converts the input speech of any individual into audiovisual streams of potentially-infinitely many output speakers.
Learning to Move with Affordance Maps
In this paper, we combine the best of both worlds with a modular approach that learns a spatial representation of a scene that is trained to be effective when coupled with traditional geometric planners.
Hierarchical Deep Stereo Matching on High-resolution Images
We explore the problem of real-time stereo matching on high-res imagery.
Inferring Distributions Over Depth from a Single Image
Such approaches cannot diagnose when failures might occur.
What You See is What You Get: Exploiting Visibility for 3D Object Detection
On the NuScenes 3D detection benchmark, we show that, by adding an additional stream for visibility input, we can significantly improve the overall detection accuracy of a state-of-the-art 3D detector.
Learning to Optimally Segment Point Clouds
We prove that if we score a segmentation by the worst objectness among its individual segments, there is an efficient algorithm that finds the optimal worst-case segmentation among an exponentially large number of candidate segmentations.
Volumetric Correspondence Networks for Optical Flow
As a result, SOTA networks also employ various heuristics designed to limit volumetric processing, leading to limited accuracy and overfitting.
Ranked #11 on
Optical Flow Estimation
on Sintel-final
Are we asking the right questions in MovieQA?
Joint vision and language tasks like visual question answering are fascinating because they explore high-level understanding, but at the same time, can be more prone to language biases.
Argoverse: 3D Tracking and Forecasting with Rich Maps
In our baseline experiments, we illustrate how detailed map information such as lane direction, driveable area, and ground height improves the accuracy of 3D object tracking and motion forecasting.
Learning to Track Any Object
Moreover, at test time the same network can be applied to detection and tracking, resulting in a unified approach for the two tasks.
MetaPix: Few-Shot Video Retargeting
We address the task of unsupervised retargeting of human actions from one video to another.
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning
In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved.
Weakly-supervised Action Localization with Background Modeling
Our approach makes use of two innovations to attention-modeling in weakly-supervised learning.
Action Localization
Weakly Supervised Action Localization
+1
Growing a Brain: Fine-Tuning by Increasing Model Capacity
One of their remarkable properties is the ability to transfer knowledge from a large source dataset to a (typically smaller) target dataset.
Shapes and Context: In-the-Wild Image Synthesis & Manipulation
We introduce a data-driven approach for interactively synthesizing in-the-wild images from semantic label maps.
Do Image Classifiers Generalize Across Time?
Additionally, we evaluate three detection models and show that natural perturbations induce both classification as well as localization errors, leading to a median drop in detection mAP of 14 points.
A Systematic Framework for Natural Perturbations from Videos
We introduce a systematic framework for quantifying the robustness of classifiers to naturally occurring perturbations of images found in videos.
Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
We also revisit existing approaches for fast convergence and show that budget-aware learning schedules readily outperform such approaches under (the practical but under-explored) budgeted training setting.
Towards Segmenting Anything That Moves
To address this concern, we propose two new benchmarks for generic, moving object detection, and show that our model matches top-down methods on common categories, while significantly out-performing both top-down and bottom-up methods on never-before-seen categories.
DistInit: Learning Video Representations Without a Single Labeled Video
In this work, we propose an alternative approach to learning video representations that require no semantically labeled videos and instead leverages the years of effort in collecting and labeling large and clean still-image datasets.
Ranked #69 on
Action Recognition
on HMDB-51
(using extra training data)
Photo-Sketching: Inferring Contour Drawings from Images
Edges, boundaries and contours are important subjects of study in both computer graphics and computer vision.
Online Model Distillation for Efficient Video Inference
Rather than learn a specialized student model on offline data from the video stream, we train the student in an online fashion on the live video, intermittently running the teacher to provide a target for learning.
Few-Shot Human Motion Prediction via Meta-Learning
This paper addresses the problem of few-shot human motion prediction, in the spirit of the recent progress on few-shot learning and meta-learning.
Recycle-GAN: Unsupervised Video Retargeting
We introduce a data-driven approach for unsupervised video retargeting that translates content from one domain to another while preserving the style native to a domain, i. e., if contents of John Oliver's speech were to be transferred to Stephen Colbert, then the generated content/speech should be in Stephen Colbert's style.
Active Testing: An Efficient and Robust Framework for Estimating Accuracy
Much recent work on visual recognition aims to scale up learning to massive, noisily-annotated datasets.
Cross-Domain Image Matching with Deep Feature Maps
We investigate the problem of automatically determining what type of shoe left an impression found at a crime scene.
Active Learning with Partial Feedback
While many active learning papers assume that the learner can simply ask for a label and receive it, real annotation often presents a mismatch between the form of a label (say, one among many classes), and the form of an annotation (typically yes/no binary feedback).
Brute-Force Facial Landmark Analysis With A 140,000-Way Classifier
While most prior work treats this as a regression problem, we instead formulate it as a discrete $K$-way classification task, where a classifier is trained to return one of $K$ discrete alignments.
Learning to Model the Tail
We cast this problem as transfer learning, where knowledge from the data-rich classes in the head of the distribution is transferred to the data-poor classes in the tail.
Patch Correspondences for Interpreting Pixel-level CNNs
We present compositional nearest neighbors (CompNN), a simple approach to visually interpreting distributed representations learned by a convolutional neural network (CNN) for pixel-level tasks (e. g., image synthesis and segmentation).
Attentional Pooling for Action Recognition
We introduce a simple yet surprisingly powerful model to incorporate attention in action recognition and human object interaction tasks.
Ranked #7 on
Human-Object Interaction Detection
on HICO
PixelNN: Example-based Image Synthesis
We present a simple nearest-neighbor (NN) approach that synthesizes high-frequency photorealistic images from an "incomplete" signal such as a low-resolution image, a surface normal map, or edges.
Learning Policies for Adaptive Tracking with Deep Feature Cascades
Our fundamental insight is to take an adaptive approach, where easy frames are processed with cheap features (such as pixel values), while challenging frames are processed with invariant but expensive deep features.
Unconstrained Face Detection and Open-Set Face Recognition Challenge
Face detection and recognition benchmarks have shifted toward more difficult environments.
Comparing Apples and Oranges: Off-Road Pedestrian Detection on the NREC Agricultural Person-Detection Dataset
Person detection from vehicles has made rapid progress recently with the advent of multiple highquality datasets of urban and highway driving, yet no large-scale benchmark is available for the same problem in off-road or agricultural environments.
Tracking as Online Decision-Making: Learning a Policy from Streaming Videos with Reinforcement Learning
We formulate tracking as an online decision-making process, where a tracking agent must follow an object despite ambiguous image frames and a limited computational budget.
Predictive-Corrective Networks for Action Detection
While deep feature learning has revolutionized techniques for static-image understanding, the same does not quite hold for video processing.
ActionVLAD: Learning spatio-temporal aggregation for action classification
In this work, we introduce a new video representation for action classification that aggregates local convolutional features across the entire spatio-temporal extent of the video.
Expecting the Unexpected: Training Detectors for Unusual Pedestrians with Adversarial Imposters
Such "in-the-tail" data is notoriously hard to observe, making both training and testing difficult.
Need for Speed: A Benchmark for Higher Frame Rate Object Tracking
In this paper, we propose the first higher frame rate video dataset (called Need for Speed - NfS) and benchmark for visual object tracking.
PixelNet: Representation of the pixels, by the pixels, and for the pixels
We explore design principles for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation.
3D Human Pose Estimation = 2D Pose Estimation + Matching
While many approaches try to directly predict 3D pose from image measurements, we explore a simple architecture that reasons through intermediate 2D pose predictions.
Ranked #248 on
3D Human Pose Estimation
on Human3.6M
Tinkering Under the Hood: Interactive Zero-Shot Learning with Net Surgery
We consider the task of visual net surgery, in which a CNN can be reconfigured without extra data to recognize novel concepts that may be omitted from the training set.
Finding Tiny Faces
We explore three aspects of the problem in the context of finding small faces: the role of scale invariance, image resolution, and contextual reasoning.
Ranked #14 on
Face Detection
on WIDER Face (Medium)
PixelNet: Towards a General Pixel-level Architecture
We explore architectures for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation.
The Open World of Micro-Videos
Micro-videos are six-second videos popular on social media networks with several unique properties.
Depth-Based Hand Pose Estimation: Data, Methods, and Challenges
To spur further progress we introduce a challenging new dataset with diverse, cluttered scenes.
Look and Think Twice: Capturing Top-Down Visual Attention With Feedback Convolutional Neural Networks
While feedforward deep convolutional neural networks (CNNs) have been a great success in computer vision, it is important to remember that the human visual contex contains generally more feedback connections than foward connections.
Understanding Everyday Hands in Action From RGB-D Images
We analyze functional manipulations of handheld objects, formalizing the problem as one of fine-grained grasp classification.
Bottom-Up and Top-Down Reasoning with Hierarchical Rectified Gaussians
We show that RGs can be optimized with a quadratic program (QP), that can in turn be optimized with a recurrent neural network (with rectified linear units).
Ranked #40 on
Pose Estimation
on MPII Human Pose
First-Person Pose Recognition Using Egocentric Workspaces
In egocentric views, hands and arms are observable within a well defined volume in front of the camera.
Multi-scale recognition with DAG-CNNs
We explore multi-scale convolutional neural nets (CNNs) for image classification.
Depth-based hand pose estimation: methods, data, and challenges
To spur further progress we introduce a challenging new dataset with diverse, cluttered scenes.
Do We Need More Training Data?
Datasets for training object recognition systems are steadily increasing in size.
Egocentric Pose Recognition in Four Lines of Code
We tackle the problem of estimating the 3D pose of an individual's upper limbs (arms+hands) from a chest mounted depth-camera.
3D Hand Pose Detection in Egocentric RGB-D Images
We focus on the task of everyday hand pose estimation from egocentric viewpoints.
Capturing Long-tail Distributions of Object Subcategories
We argue that object subcategories follow a long-tail distribution: a few subcategories are common, while many are rare.
Parsing Videos of Actions with Segmental Grammars
Real-world videos of human activities exhibit temporal structure at various scales; long videos are typically composed out of multiple action instances, where each instance is itself composed of sub-actions with variable durations and orderings.
Analysis by Synthesis: 3D Object Recognition by Object Reconstruction
We introduce an efficient "brute-force" approach to inference that searches through a large number of candidate reconstructions, returning the optimal one.
Parsing Occluded People
Occlusion poses a significant difficulty for object recognition due to the combinatorial diversity of possible occlusion patterns.
Microsoft COCO: Common Objects in Context
We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding.
Dual coordinate solvers for large-scale structural SVMs
This manuscript describes a method for training linear SVMs (including binary SVMs, SVM regression, and structural SVMs) from large, out-of-core training datasets.
Exploring Weak Stabilization for Motion Feature Extraction
We describe novel but simple motion features for the problem of detecting objects in video sequences.
Histograms of Sparse Codes for Object Detection
Object detection has seen huge progress in recent years, much thanks to the heavily-engineered Histograms of Oriented Gradients (HOG) features.
Self-Paced Learning for Long-Term Tracking
We address the problem of long-term object tracking, where the object may become occluded or leave-the-view.
Analyzing 3D Objects in Cluttered Images
We use a morphable model to capture 3D within-class variation, and use a weak-perspective camera model to capture viewpoint.
Video Annotation and Tracking with Active Learning
We introduce a novel active learning framework for video annotation.
Statistical Tests for Optimization Efficiency
Learning problems such as logistic regression are typically formulated as pure optimization problems defined on some loss function.
Bilinear classifiers for visual recognition
Bilinear classifiers are a discriminative variant of bilinear models, which capture the dependence of data on multiple factors.
