Search Results for author:
Found 175 papers, 83 papers with code
GroupViT: Semantic Segmentation Emerges from Text Supervision
With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i. e., without any further fine-tuning.
Global Context Vision Transformers
Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation using MS COCO and ADE20K datasets outperform prior work consistently.
Ranked #132 on
Semantic Segmentation
on ADE20K
Multimodal Unsupervised Image-to-Image Translation
To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain.
Multimodal Unsupervised Image-To-Image Translation
Translation
+1
Unsupervised Image-to-Image Translation Networks
Unsupervised image-to-image translation aims at learning a joint distribution of images in different domains by using images from the marginal distributions in individual domains.
Domain Adaptation
Multimodal Unsupervised Image-To-Image Translation
+2
A Closed-form Solution to Photorealistic Image Stylization
Photorealistic image stylization concerns transferring style of a reference photo to a content photo with the constraint that the stylized photo should remain photorealistic.
Loss Functions for Neural Networks for Image Processing
Neural networks are becoming central in several areas of computer vision and image processing and different architectures have been proposed to solve specific problems.
Video-to-Video Synthesis
We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e. g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
We present a new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs).
Ranked #2 on
Sketch-to-Image Translation
on COCO-Stuff
Conditional Image Generation
Fundus to Angiography Generation
+5
Joint Discriminative and Generative Learning for Person Re-identification
To this end, we propose a joint learning framework that couples re-id learning and data generation end-to-end.
Ranked #1 on
Person Re-Identification
on UAV-Human
Image-to-Image Translation
Unsupervised Domain Adaptation
+1
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
Finally, the two input images are warped and linearly fused to form each intermediate frame.
VILA: On Pre-training for Visual Language Models
Visual language models (VLMs) rapidly progressed with the recent success of large language models.
Ranked #21 on
Visual Question Answering
on MM-Vet
Few-shot Video-to-Video Synthesis
To address the limitations, we propose a few-shot vid2vid framework, which learns to synthesize videos of previously unseen subjects or scenes by leveraging few example images of the target at test time.
Ranked #1 on
Video-to-Video Synthesis
on YouTube Dancing
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
It then uses the warped features and features of the first image to construct a cost volume, which is processed by a CNN to estimate the optical flow.
Ranked #3 on
Dense Pixel Correspondence Estimation
on HPatches
Dense Pixel Correspondence Estimation
Optical Flow Estimation
Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation
We investigate two crucial and closely related aspects of CNNs for optical flow estimation: models and training.
Ranked #7 on
Optical Flow Estimation
on KITTI 2012
A Fusion Approach for Multi-Frame Optical Flow Estimation
To date, top-performing optical flow estimation methods only take pairs of consecutive frames into account.
Few-Shot Unsupervised Image-to-Image Translation
Unsupervised image-to-image translation methods learn to map images in a given class to an analogous image in a different class, drawing on unstructured (non-registered) datasets of images.
NVAE: A Deep Hierarchical Variational Autoencoder
For example, on CIFAR-10, NVAE pushes the state-of-the-art from 2. 98 to 2. 91 bits per dimension, and it produces high-quality images on CelebA HQ.
Ranked #3 on
Image Generation
on FFHQ 256 x 256
(bits/dimension metric)
BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object.
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
We present FoundationPose, a unified foundation model for 6D object pose estimation and tracking, supporting both model-based and model-free setups.
Fast and Accurate Point Cloud Registration using Trees of Gaussian Mixtures
Point cloud registration sits at the core of many important and challenging 3D perception problems including autonomous navigation, SLAM, object/scene recognition, and augmented reality.
FasterViT: Fast Vision Transformers with Hierarchical Attention
At a high level, global self-attentions enable the efficient cross-window communication at lower costs.
Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU
We introduce a hybrid CPU/GPU version of the Asynchronous Advantage Actor-Critic (A3C) algorithm, currently the state-of-the-art method in reinforcement learning for various gaming tasks.
DeepGMR: Learning Latent Gaussian Mixture Models for Registration
Point cloud registration is a fundamental problem in 3D computer vision, graphics and robotics.
MoCoGAN: Decomposing Motion and Content for Video Generation
The proposed framework generates a video by mapping a sequence of random vectors to a sequence of video frames.
Synthetically Trained Neural Networks for Learning Human-Readable Plans from Real-World Demonstrations
We present a system to infer and execute a human-readable program from a real-world demonstration.
Robotics
FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation
This technical report summarizes the winning solution for the 3D Occupancy Prediction Challenge, which is held in conjunction with the CVPR 2023 Workshop on End-to-End Autonomous Driving and CVPR 23 Workshop on Vision-Centric Autonomous Driving Workshop.
Ranked #1 on
Prediction Of Occupancy Grid Maps
on Occ3D-nuScenes
PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image
This paper proposes a deep neural architecture, PlaneRCNN, that detects and reconstructs piecewise planar surfaces from a single RGB image.
Dancing to Music
In the analysis phase, we decompose a dance into a series of basic dance units, through which the model learns how to move.
Ranked #3 on
Motion Synthesis
on BRACE
Pixel-Adaptive Convolutional Neural Networks
In addition, we also demonstrate that PAC can be used as a drop-in replacement for convolution layers in pre-trained networks, resulting in consistent performance improvements.
Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network.
Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting
We introduce a novel method to obtain high-quality 3D reconstructions from consumer RGB-D sensors.
Learning Linear Transformations for Fast Arbitrary Style Transfer
Recent arbitrary style transfer methods transfer second order statistics from reference image onto content image via a multiplication between content image features and a transformation matrix, which is computed from features with a pre-determined algorithm.
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals
Official Torch7 implementation of "V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map", CVPR 2018
Ranked #5 on
Hand Pose Estimation
on HANDS 2017
Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection
Weakly supervised learning has emerged as a compelling tool for object detection by reducing the need for strong supervision during training.
Ranked #1 on
Weakly Supervised Object Detection
on COCO test-dev
UFO²: A Unified Framework towards Omni-supervised Object Detection
Existing work on object detection often relies on a single form of annotation: the model is trained using either accurate yet costly bounding boxes or cheaper but less expressive image-level tags.
Superpixel Sampling Networks
Superpixels provide an efficient low/mid-level representation of image data, which greatly reduces the number of image primitives for subsequent vision tasks.
Geometry-Aware Learning of Maps for Camera Localization
Maps are a key component in image-based camera localization and visual SLAM systems: they are used to establish geometric constraints between images, correct drift in relative pose estimation, and relocalize cameras after lost tracking.
Ranked #5 on
Visual Localization
on Oxford RobotCar Full
GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras
Since the joint reconstruction of human motions and camera poses is underconstrained, we propose a global trajectory predictor that generates global human trajectories based on local body movements.
Ranked #1 on
Global 3D Human Pose Estimation
on EMDB
Score-based Generative Modeling in Latent Space
Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling.
Ranked #3 on
Image Generation
on CIFAR-10
(FD metric)
See through Gradients: Image Batch Recovery via GradInversion
In this work, we introduce GradInversion, using which input images from a larger batch (8 - 48 images) can also be recovered for large networks such as ResNets (50 layers), on complex datasets such as ImageNet (1000 classes, 224x224 px).
DiffiT: Diffusion Vision Transformers for Image Generation
In this paper, we study the effectiveness of ViTs in diffusion-based generative learning and propose a new model denoted as Diffusion Vision Transformers (DiffiT).
Ranked #4 on
Image Generation
on ImageNet 256x256
FreeSOLO: Learning to Segment Objects without Annotations
FreeSOLO further demonstrates superiority as a strong pre-training method, outperforming state-of-the-art self-supervised pre-training methods by +9. 8% AP when fine-tuning instance segmentation with only 5% COCO masks.
Few-Shot Adaptive Gaze Estimation
Inter-personal anatomical differences limit the accuracy of person-independent gaze estimation networks.
Ranked #1 on
Gaze Estimation
on MPII Gaze
(using extra training data)
Importance Estimation for Neural Network Pruning
On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0. 02% in the top-1 accuracy on ImageNet.
Neural RGB->D Sensing: Depth and Uncertainty from a Video Camera
Depth sensing is crucial for 3D reconstruction and scene understanding.
SPLATNet: Sparse Lattice Networks for Point Cloud Processing
We present a network architecture for processing point clouds that directly operates on a collection of points represented as a sparse set of samples in a high-dimensional lattice.
Ranked #29 on
Semantic Segmentation
on ScanNet
STEP: Spatio-Temporal Progressive Learning for Video Action Detection
In this paper, we propose Spatio-TEmporal Progressive (STEP) action detector---a progressive learning framework for spatio-temporal action detection in videos.
Ranked #7 on
Action Detection
on UCF101-24
Self-supervised Single-view 3D Reconstruction via Semantic Consistency
To the best of our knowledge, we are the first to try and solve the single-view reconstruction problem without a category-specific template mesh or semantic keypoints.
SCOPS: Self-Supervised Co-Part Segmentation
Parts provide a good intermediate representation of objects that is robust with respect to the camera, pose and appearance variations.
Ranked #4 on
Unsupervised Keypoint Estimation
on CUB
Self-Supervised Viewpoint Learning From Image Collections
Training deep neural networks to estimate the viewpoint of objects requires large labeled training datasets.
Switchable Temporal Propagation Network
Our approach is based on a temporal propagation network (TPN), which models the transition-related affinity between a pair of frames in a purely data-driven manner.
Joint-task Self-supervised Learning for Temporal Correspondence
Our learning process integrates two highly related tasks: tracking large image regions \emph{and} establishing fine-grained pixel-level associations between consecutive video frames.
Bi3D: Stereo Depth Estimation via Binary Classifications
Given a strict time budget, Bi3D can detect objects closer than a given distance in as little as a few milliseconds, or estimate depth with arbitrarily coarse quantization, with complexity linear with the number of quantization levels.
Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems.
Pruning Convolutional Neural Networks for Resource Efficient Inference
We propose a new criterion based on Taylor expansion that approximates the change in the cost function induced by pruning network parameters.
DexYCB: A Benchmark for Capturing Hand Grasping of Objects
We introduce DexYCB, a new dataset for capturing hand grasping of objects.
AdaViT: Adaptive Tokens for Efficient Vision Transformer
A-ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
Ranked #34 on
Efficient ViTs
on ImageNet-1K (with DeiT-S)
Simultaneous Edge Alignment and Learning
Edge detection is among the most fundamental vision problems for its role in perceptual grouping and its wide applications.
Convolutional Tensor-Train LSTM for Spatio-temporal Learning
Learning from spatio-temporal data has numerous applications such as human-behavior analysis, object tracking, video compression, and physics simulation. However, existing methods still perform poorly on challenging video tasks such as long-term forecasting.
Ranked #1 on
Video Prediction
on KTH
(Cond metric)
Unsupervised Video Interpolation Using Cycle Consistency
We further introduce a pseudo supervised loss term that enforces the interpolated frames to be consistent with predictions of a pre-trained interpolation model.
Ranked #1 on
Video Frame Interpolation
on UCF101
(PSNR (sRGB) metric)
LITA: Language Instructed Temporal-Localization Assistant
In addition to leveraging existing video datasets with timestamps, we propose a new task, Reasoning Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for learning and evaluating this task.
Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?
We initially observed that the nuScenes dataset, characterized by relatively simple driving scenarios, leads to an under-utilization of perception information in end-to-end models incorporating ego status, such as the ego vehicle's velocity.
Context-Aware Synthesis and Placement of Object Instances
Learning to insert an object instance into an image in a semantically coherent manner is a challenging and interesting problem.
Joint Disentangling and Adaptation for Cross-Domain Person Re-Identification
To this end, we propose a joint learning framework that disentangles id-related/unrelated features and enforces adaptation to work on the id-related feature space exclusively.
Ranked #6 on
Unsupervised Domain Adaptation
on Market to MSMT
Two-shot Spatially-varying BRDF and Shape Estimation
Extensive experiments on both synthetic and real-world datasets show that our network trained on a synthetic dataset can generalize well to real-world images.
SENSE: a Shared Encoder Network for Scene-flow Estimation
We introduce a compact network for holistic scene flow estimation, called SENSE, which shares common encoder features among four closely-related tasks: optical flow estimation, disparity estimation from stereo, occlusion estimation, and semantic segmentation.
Extreme View Synthesis
We present Extreme View Synthesis, a solution for novel view extrapolation that works even when the number of input images is small--as few as two.
AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One
A handful of visual foundation models (VFMs) have recently emerged as the backbones for numerous downstream tasks.
Contrastive Learning for Weakly Supervised Phrase Grounding
Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions.
UNAS: Differentiable Architecture Search Meets Reinforcement Learning
Our framework brings the best of both worlds, and it enables us to search for architectures with both differentiable and non-differentiable criteria in one unified framework while maintaining a low search cost.
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
VAEBM captures the overall mode structure of the data distribution using a state-of-the-art VAE and it relies on its EBM component to explicitly exclude non-data-like regions from the model and refine the image samples.
Ranked #1 on
Image Generation
on Stacked MNIST
Binary TTC: A Temporal Geofence for Autonomous Navigation
Time-to-contact (TTC), the time for an object to collide with the observer's plane, is a powerful tool for path planning: it is potentially more informative than the depth, velocity, and acceleration of objects in the scene -- even for humans.
A Variational Perspective on Solving Inverse Problems with Diffusion Models
To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution.
Meshlet Priors for 3D Mesh Reconstruction
Meshlets act as a dictionary of local features and thus allow to use learned priors to reconstruct object meshes in any pose and from unseen classes, even when the noise is large and the samples sparse.
Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Models
Our Landmark DEQ (LDEQ) achieves state-of-the-art performance on the challenging WFLW facial landmark dataset, reaching $3. 92$ NME with fewer parameters and a training memory cost of $\mathcal{O}(1)$ in the number of recurrent modules.
Ranked #2 on
Face Alignment
on WFLW
Weakly-Supervised Physically Unconstrained Gaze Estimation
A major challenge for physically unconstrained gaze estimation is acquiring training data with 3D gaze annotations for in-the-wild and outdoor scenarios.
Ranked #3 on
Gaze Estimation
on Gaze360
The Best Defense is a Good Offense: Adversarial Augmentation against Adversarial Attacks
Many defenses against adversarial attacks (\eg robust classifiers, randomization, or image purification) use countermeasures put to work only after the attack has been crafted.
CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs
We represent the correspondence maps of different images as warped coordinate frames transformed from a canonical coordinate frame, i. e., the correspondence map, which describes the structure (e. g., the shape of a face), is controlled via a transformation.
Discovering Nonlinear Relations with Minimum Predictive Information Regularization
Identifying the underlying directional relations from observational time series with nonlinear interactions and complex relational structures is key to a wide range of applications, yet remains a hard problem.
SMRD: SURE-based Robust MRI Reconstruction with Diffusion Models
However, diffusion models require careful tuning of inference hyperparameters on a validation set and are still sensitive to distribution shifts during testing.
ViR: Towards Efficient Vision Retention Backbones
Inspired by this trend, we propose a new class of computer vision models, dubbed Vision Retention Networks (ViR), with dual parallel and recurrent formulations, which strike an optimal balance between fast inference and parallel training with competitive performance.
Zero-shot Pose Transfer for Unrigged Stylized 3D Characters
We present a zero-shot approach that requires only the widely available deformed non-stylized avatars in training, and deforms stylized characters of significantly different shapes at inference.
Global Vision Transformer Pruning with Hessian-Aware Saliency
This work aims on challenging the common design philosophy of the Vision Transformer (ViT) model with uniform dimension across all the stacked blocks in a model stage, where we redistribute the parameters both across transformer blocks and between different structures within the block via the first systematic attempt on global structural pruning.
Improving Landmark Localization with Semi-Supervised Learning
First, we propose the framework of sequential multitasking and explore it here through an architecture for landmark localization where training with class labels acts as an auxiliary signal to guide the landmark localization on unlabeled data.
Ranked #41 on
Face Alignment
on 300W
Budget-Aware Activity Detection with A Recurrent Policy Network
In this paper, we address the challenging problem of efficient temporal activity detection in untrimmed long videos.
IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification
Deep residual networks (ResNets) made a recent breakthrough in deep learning.
Hand Pose Estimation via Latent 2.5D Heatmap Regression
Estimating the 3D pose of a hand is an essential part of human-computer interaction.
Light-weight Head Pose Invariant Gaze Tracking
Unconstrained remote gaze tracking using off-the-shelf cameras is a challenging problem.
A Lightweight Approach for On-the-Fly Reflectance Estimation
In addition, we have created a large synthetic dataset, SynBRDF, which comprises a total of $500$K RGBD images rendered with a physically-based ray tracer under a variety of natural illumination, covering $5000$ materials and $5000$ shapes.
Deep Semantic Face Deblurring
In this paper, we present an effective and efficient face deblurring algorithm by exploiting semantic cues via deep convolutional neural networks (CNNs).
Reblur2Deblur: Deblurring Videos via Self-Supervised Learning
Motion blur is a fundamental problem in computer vision as it impacts image quality and hinders inference.
Learning Adaptive Parameter Tuning for Image Processing
The non-stationary nature of image characteristics calls for adaptive processing, based on the local image content.
Learning Binary Residual Representations for Domain-specific Video Streaming
Specifically, we target a streaming setting where the videos to be streamed from a server to a client are all in the same domain and they have to be compressed to a small size for low-latency transmission.
Separating Reflection and Transmission Images in the Wild
The reflections caused by common semi-reflectors, such as glass windows, can impact the performance of computer vision algorithms.
On Nearest Neighbors in Non Local Means Denoising
To denoise a reference patch, the Non-Local-Means denoising filter processes a set of neighbor patches.
Cascaded Scene Flow Prediction using Semantic Segmentation
Given two consecutive frames from a pair of stereo cameras, 3D scene flow methods simultaneously estimate the 3D geometry and motion of the observed scene.
Multiframe Scene Flow with Piecewise Rigid Motion
We introduce a novel multiframe scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences.
Learning Affinity via Spatial Propagation Networks
Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be the output from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix.
Learning to Segment Instances in Videos with Spatial Propagation Network
In addition, we apply a filter on the refined score map that aims to recognize the best connected region using spatial and temporal consistencies in the video.
Deep Learning with Energy-efficient Binary Gradient Cameras
Power consumption is a critical factor for the deployment of embedded computer vision systems.
Locally Non-rigid Registration for Mobile HDR Photography
Image registration for stack-based HDR photography is challenging.
Hierarchical Subquery Evaluation for Active Learning on a Graph
Under some specific circumstances, Expected Error Reduction has been one of the strongest-performing informativeness criteria for active learning.
Domain Stylization: A Strong, Simple Baseline for Synthetic to Real Image Domain Adaptation
Deep neural networks have largely failed to effectively utilize synthetic data when applied to real images due to the covariate shift problem.
Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset
Scene motion, multiple reflections, and sensor noise introduce artifacts in the depth reconstruction performed by time-of-flight cameras.
EOE: Expected Overlap Estimation over Unstructured Point Cloud Data
We present an iterative overlap estimation technique to augment existing point cloud registration algorithms that can achieve high performance in difficult real-world situations where large pose displacement and non-overlapping geometry would otherwise cause traditional methods to fail.
Learning Superpixels With Segmentation-Aware Affinity Loss
Specifically, we propose a new loss function that takes the segmentation error into account for affinity learning.
Making Convolutional Networks Recurrent for Visual Sequence Learning
Recurrent neural networks (RNNs) have emerged as a powerful model for a broad range of machine learning problems that involve sequential data.
Neural Causal Discovery with Learnable Input Noise
Learning causal relations from observational time series with nonlinear interactions and complex causal structures is a key component of human intelligence, and has a wide range of applications.
Neural Inverse Rendering of an Indoor Scene from a Single Image
Inverse rendering aims to estimate physical attributes of a scene, e. g., reflectance, geometry, and lighting, from image(s).
NRMVS: Non-Rigid Multi-View Stereo
We show that creating a dense 4D structure from a few RGB images with non-rigid changes is possible, and demonstrate that our method can be used to interpolate novel deformed scenes from various combinations of these deformation estimates derived from the sparse views.
Fully-Connected CRFs with Non-Parametric Pairwise Potential
Conditional Random Fields (CRFs) are used for diverse tasks, ranging from image denoising to object recognition.
Modeling Object Appearance Using Context-Conditioned Component Analysis
In this work we remove the image space alignment limitations of existing subspace models by conditioning the models on a shape dependent context that allows for the complex, non-linear structure of the appearance of the visual object to be captured and shared.
Online Detection and Classification of Dynamic Hand Gestures With Recurrent 3D Convolutional Neural Network
Automatic detection and classification of dynamic hand gestures in real-world systems intended for human computer interaction is challenging as: 1) there is a large diversity in how people perform gestures, making detection and classification difficult; 2) the system must work online in order to avoid noticeable lag between performing a gesture and its classification; in fact, a negative lag (classification before the gesture is finished) is desirable, as feedback to the user can then be truly instantaneous.
Accelerated Generative Models for 3D Point Cloud Data
In this paper we introduce a method for constructing compact generative representations of PCD at multiple levels of detail.
Dynamic Facial Analysis: From Bayesian Filtering to Recurrent Neural Network
We are inspired by the fact that the computation performed in an RNN bears resemblance to Bayesian filters, which have been used for tracking in many previous methods for facial analysis from videos.
Ranked #1 on
Head Pose Estimation
on BIWI
(MAE (trained with BIWI data) metric, using extra
training data)
Polarimetric Multi-View Stereo
Multi-view stereo relies on feature correspondences for 3D reconstruction, and thus is fundamentally flawed in dealing with featureless scenes.
Robust Model-Based 3D Head Pose Estimation
We introduce a method for accurate three dimensional head pose estimation using a commodity depth camera.
Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments
In order to predict valid affordances and learn possible 3D human poses in indoor scenes, we need to understand the semantic and geometric structure of a scene as well as its potential interactions with a human.
Towards annotation-efficient segmentation via image-to-image translation
Specifically, we propose a semi-supervised framework that employs unpaired image-to-image translation between two domains, presence vs. absence of cancer, as the unsupervised objective.
Few-Shot Viewpoint Estimation
Through extensive experimentation on the ObjectNet3D and Pascal3D+ benchmark datasets, we demonstrate that our framework, which we call MetaView, significantly outperforms fine-tuning the state-of-the-art models with few examples, and that the specific architectural innovations of our method are crucial to achieving good performance.
Video Stitching for Linear Camera Arrays
Despite the long history of image and video stitching research, existing academic and commercial solutions still produce strong artifacts.
Learning Propagation for Arbitrarily-structured Data
We experiment with semantic segmentation networks, where we use our propagation module to jointly train on different data -- images, superpixels and point clouds.
Angular Visual Hardness
We also find that AVH has a statistically significant correlation with human visual hardness.
Exploiting Semantics for Face Image Deblurring
Specifically, we first use a coarse deblurring network to reduce the motion blur on the input face image.
Learning to Generate Multiple Style Transfer Outputs for an Input Sentence
Specifically, we decompose the latent representation of the input sentence to a style code that captures the language style variation and a content code that encodes the language style-independent content.
Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild
One major challenge for monocular 3D human pose estimation in-the-wild is the acquisition of training data that contains unconstrained images annotated with accurate 3D poses.
Monocular 3D Human Pose Estimation
Weakly-superavised 3D Human Pose Estimation
+1
Gaze-Sensing LEDs for Head Mounted Displays
We introduce a new gaze tracker for Head Mounted Displays (HMDs).
Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints
Estimating 3D hand pose from 2D images is a difficult, inverse problem due to the inherent scale and depth ambiguities.
Ranked #10 on
3D Hand Pose Estimation
on DexYCB
Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera
Our insight is that although its scale and quality are inconsistent with other views, the depth estimation from a single view can be used to reason about the globally coherent geometry of dynamic contents.
How to Close Sim-Real Gap? Transfer with Segmentation!
Combining the control policy learned from simulation with the perception model, we achieve an impressive $\bf{88\%}$ success rate in grasping a tiny sphere with a real robot.
Robotics
Hierarchical Contrastive Motion Learning for Video Action Recognition
Thus, the motion features at higher levels are trained to gradually capture semantic dynamics and evolve more discriminative for action recognition.
Improving Deep Stereo Network Generalization with Geometric Priors
End-to-end deep learning methods have advanced stereo vision in recent years and obtained excellent results when the training and test data are similar.
A Contrastive Learning Approach for Training Variational Autoencoder Priors
To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior.
Ranked #6 on
Image Generation
on CelebA 256x256
(FID metric)
UFO$^2$: A Unified Framework towards Omni-supervised Object Detection
Existing work on object detection often relies on a single form of annotation: the model is trained using either accurate yet costly bounding boxes or cheaper but less expressive image-level tags.
Displacement-Invariant Cost Computation for Efficient Stereo Matching
A common way to speed up the computation is to downsample the feature volume, but this loses high-frequency details.
Online Adaptation for Consistent Mesh Reconstruction in the Wild
This paper presents an algorithm to reconstruct temporally consistent 3D meshes of deformable object instances from videos in the wild.
Parameter Efficient Multimodal Transformers for Video Representation Learning
The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model.
Neural 3D Clothes Retargeting from a Single Image
In this paper, we present a method of clothes retargeting; generating the potential poses and deformations of a given 3D clothing template model to fit onto a person in a single RGB image.
Learning Affinity via Spatial Propagation Network
Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be the output from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix.
Learning to Track Instances without Video Annotations
Tracking segmentation masks of multiple instances has been intensively studied, but still faces two fundamental challenges: 1) the requirement of large-scale, frame-wise annotation, and 2) the complexity of two-stage approaches.
KAMA: 3D Keypoint Aware Body Mesh Articulation
We present KAMA, a 3D Keypoint Aware Mesh Articulation approach that allows us to estimate a human body mesh from the positions of 3D body keypoints.
Ranked #47 on
3D Human Pose Estimation
on 3DPW
Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation
Hand pose estimation is difficult due to different environmental conditions, object- and self-occlusion as well as diversity in hand shape and appearance.
Self-Supervised Learning on 3D Point Clouds by Learning Discrete Generative Models
In this work, we introduce a general method for 3D self-supervised representation learning that 1) remains agnostic to the underlying neural network architecture, and 2) specifically leverages the geometric nature of 3D point cloud data.
Privacy Vulnerability of Split Computing to Data-Free Model Inversion Attacks
Prior works usually assume that SC offers privacy benefits as only intermediate features, instead of private data, are shared from devices to the cloud.
LANA: Latency Aware Network Acceleration
We analyze three popular network architectures: EfficientNetV1, EfficientNetV2 and ResNeST, and achieve accuracy improvement for all models (up to $3. 0\%$) when compressing larger models to the latency level of smaller models.
Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting
In this paper, we propose a unified, learning-based inverse rendering framework that formulates 3D spatially-varying lighting.
Learning Contrastive Representation for Semantic Correspondence
Dense correspondence across semantically related images has been extensively studied, but still faces two challenges: 1) large variations in appearance, scale and pose exist even for objects from the same category, and 2) labeling pixel-level dense correspondences is labor intensive and infeasible to scale.
Hardware-Aware Network Transformation
In the second phase, it solves the combinatorial selection of efficient operations using a novel constrained integer linear optimization approach.
Self-Supervised Object Detection via Generative Image Synthesis
We present SSOD, the first end-to-end analysis-by synthesis framework with controllable GANs for the task of self-supervised object detection.
Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
It is therefore interesting to study how these two tasks can be coupled to benefit each other.
Convolutional Tensor-Train LSTM for Long-Term Video Prediction
Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames. Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations.
Long History Short-Term Memory for Long-Term Video Prediction
While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging — ambiguous future or error propagation over time yield blurry predictions.
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors
To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior.
Learning Continuous Environment Fields via Implicit Functions
We propose a novel scene representation that encodes reaching distance -- the distance between any position in the scene to a goal along a feasible trajectory.
Federated Learning with Heterogeneous Architectures using Graph HyperNetworks
Standard Federated Learning (FL) techniques are limited to clients with identical network architectures.
Do Gradient Inversion Attacks Make Federated Learning Unsafe?
Federated learning (FL) allows the collaborative training of AI models without needing to share raw data.
Physics Informed RNN-DCT Networks for Time-Dependent Partial Differential Equations
This efficiently and flexibly produces a compressed representation which is used for additional conditioning of physics-informed models.
GradViT: Gradient Inversion of Vision Transformers
In this work we demonstrate the vulnerability of vision transformers (ViTs) to gradient-based inversion attacks.
DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars
In this work, we present, DRaCoN, a framework for learning full-body volumetric avatars which exploits the advantages of both the 2D and 3D neural rendering techniques.
RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis
We present a large-scale synthetic dataset for novel view synthesis consisting of ~300k images rendered from nearly 2000 complex scenes using high-quality ray tracing at high resolution (1600 x 1600 pixels).
Ranked #1 on
Novel View Synthesis
on RTMV
Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion
In this work, we propose a neural approach that estimates the 5D HDR light field from a single image, and a differentiable object insertion formulation that enables end-to-end training with image-based losses that encourage realism.
Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation
An effective approach is to supervise the training of deep neural networks with a high-fidelity dataset of desired input-output pairs, captured with a light stage.
PhysDiff: Physics-Guided Human Motion Diffusion Model
Specifically, we propose a physics-based motion projection module that uses motion imitation in a physics simulator to project the denoised motion of a diffusion step to a physically-plausible motion.
RANA: Relightable Articulated Neural Avatars
We propose RANA, a relightable and articulated neural avatar for the photorealistic synthesis of humans under arbitrary viewpoints, body poses, and lighting.
Score-based Diffusion Models in Function Space
They consist of a forward process that perturbs input data with Gaussian white noise and a reverse process that learns a score function to generate samples by denoising.
Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization
To tackle this problem, we propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing.
Generalizable One-shot Neural Head Avatar
We present a method that reconstructs and animates a 3D head avatar from a single-view portrait image.
Heterogeneous Continual Learning
We propose a novel framework and a solution to tackle the continual learning (CL) problem with changing network architectures.
Online Overexposed Pixels Hallucination in Videos with Adaptive Reference Frame Selection
We present a learning-based system to reduce these artifacts without resorting to complex acquisition mechanisms like alternating exposures or costly processing that are typical of high dynamic range (HDR) imaging.
3D Reconstruction with Generalizable Neural Fields using Scene Priors
NFP not only demonstrates SOTA scene reconstruction performance and efficiency, but it also supports single-image novel-view synthesis, which is underexplored in neural fields.
Residual Diffusion Modeling for Km-scale Atmospheric Downscaling
Predictions of weather hazard require expensive km-scale simulations driven by coarser global inputs.
PACE: Human and Camera Motion Estimation from in-the-wild Videos
This design combines the strengths of SLAM and motion priors, which leads to significant improvements in human and camera motion estimation.
COLMAP-Free 3D Gaussian Splatting
While neural rendering has led to impressive advances in scene reconstruction and novel view synthesis, it relies heavily on accurately pre-computed camera poses.
GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning
Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations.
FoVA-Depth: Field-of-View Agnostic Depth Estimation for Cross-Dataset Generalization
Unfortunately, most of the GT data is for pinhole cameras, making it impossible to properly train depth estimation models for large-FoV cameras.
