Search Results for author:
Found 40 papers, 18 papers with code
Coherent 3D Portrait Video Reconstruction via Triplane Fusion
Recent breakthroughs in single-image 3D portrait reconstruction have enabled telepresence systems to stream 3D portrait videos from a single camera in real-time, potentially democratizing telepresence.
RegionGPT: Towards Region Understanding Vision Language Model
Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the vision encoder, and the use of coarse-grained training data that lacks detailed, region-specific captions.
What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs
3D-aware Generative Adversarial Networks (GANs) have shown remarkable progress in learning to generate multi-view-consistent images and 3D geometries of scenes from collections of 2D images via neural volume rendering.
GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning
Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations.
A Unified Approach for Text- and Image-guided 4D Scene Generation
Large-scale diffusion generative models are greatly simplifying image, video and 3D asset creation from user-provided text prompts and images.
3D Reconstruction with Generalizable Neural Fields using Scene Priors
NFP not only demonstrates SOTA scene reconstruction performance and efficiency, but it also supports single-image novel-view synthesis, which is underexplored in neural fields.
Investigation of Architectures and Receptive Fields for Appearance-based Gaze Estimation
With the rapid development of deep learning technology in the past decade, appearance-based gaze estimation has attracted great attention from both computer vision and human-computer interaction research communities.
Generalizable One-shot Neural Head Avatar
We present a method that reconstructs and animates a 3D head avatar from a single-view portrait image.
Zero-shot Pose Transfer for Unrigged Stylized 3D Characters
We present a zero-shot approach that requires only the widely available deformed non-stylized avatars in training, and deforms stylized characters of significantly different shapes at inference.
Avatar Fingerprinting for Authorized Use of Synthetic Talking-Head Videos
To tackle it, we first introduce a large-scale dataset of real and synthetic videos of people interacting on a video call, where the synthetic videos are generated using the facial appearance of one person and the expressions of another.
Generative Novel View Synthesis with 3D-Aware Diffusion Models
We present a diffusion-based model for 3D-aware generative novel view synthesis from as few as a single input image.
Affordance Diffusion: Synthesizing Hand-Object Interactions
In contrast, in this work we focus on synthesizing complex interactions (ie, an articulated hand) with a given object.
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
Our approach outperforms the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks.
Ranked #2 on
Open-World Instance Segmentation
on UVO
(using extra training data)
Open Vocabulary Panoptic Segmentation
Open Vocabulary Semantic Segmentation
+4
GPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagation
In each GP Block, features are first grouped together by a fixed number of learnable group tokens; we then perform Group Propagation where global information is exchanged between the grouped features; finally, global information in the updated grouped features is returned back to the image features through a transformer decoder.
GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields
We propose GazeNeRF, a 3D-aware method for the task of gaze redirection.
CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs
We represent the correspondence maps of different images as warped coordinate frames transformed from a canonical coordinate frame, i. e., the correspondence map, which describes the structure (e. g., the shape of a face), is controlled via a transformation.
FreeSOLO: Learning to Segment Objects without Annotations
FreeSOLO further demonstrates superiority as a strong pre-training method, outperforming state-of-the-art self-supervised pre-training methods by +9. 8% AP when fine-tuning instance segmentation with only 5% COCO masks.
GroupViT: Semantic Segmentation Emerges from Text Supervision
With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i. e., without any further fine-tuning.
Efficient Geometry-aware 3D Generative Adversarial Networks
Unsupervised generation of high-quality multi-view-consistent images and 3D shapes using only collections of single-view 2D photographs has been a long-standing challenge.
Learning Continuous Environment Fields via Implicit Functions
We propose a novel scene representation that encodes reaching distance -- the distance between any position in the scene to a goal along a feasible trajectory.
Self-Supervised Object Detection via Generative Image Synthesis
We present SSOD, the first end-to-end analysis-by synthesis framework with controllable GANs for the task of self-supervised object detection.
Learning Contrastive Representation for Semantic Correspondence
Dense correspondence across semantically related images has been extensively studied, but still faces two challenges: 1) large variations in appearance, scale and pose exist even for objects from the same category, and 2) labeling pixel-level dense correspondences is labor intensive and infeasible to scale.
Weakly-Supervised Physically Unconstrained Gaze Estimation
A major challenge for physically unconstrained gaze estimation is acquiring training data with 3D gaze annotations for in-the-wild and outdoor scenarios.
Ranked #3 on
Gaze Estimation
on Gaze360
Contrastive Syn-to-Real Generalization
Training on synthetic data can be beneficial for label or data-scarce scenarios.
Learning to Track Instances without Video Annotations
Tracking segmentation masks of multiple instances has been intensively studied, but still faces two fundamental challenges: 1) the requirement of large-scale, frame-wise annotation, and 2) the complexity of two-stage approaches.
Online Adaptation for Consistent Mesh Reconstruction in the Wild
This paper presents an algorithm to reconstruct temporally consistent 3D meshes of deformable object instances from videos in the wild.
Self-Learning Transformations for Improving Gaze and Head Redirection
Furthermore, we show that in the presence of limited amounts of real-world training data, our method allows for improvements in the downstream task of semi-supervised cross-dataset gaze estimation.
Self-Supervised Viewpoint Learning From Image Collections
Training deep neural networks to estimate the viewpoint of objects requires large labeled training datasets.
Self-supervised Single-view 3D Reconstruction via Semantic Consistency
To the best of our knowledge, we are the first to try and solve the single-view reconstruction problem without a category-specific template mesh or semantic keypoints.
Content-Consistent Generation of Realistic Eyes with Style
Accurately labeled real-world training data can be scarce, and hence recent works adapt, modify or generate images to boost target datasets.
Joint-task Self-supervised Learning for Temporal Correspondence
Our learning process integrates two highly related tasks: tracking large image regions \emph{and} establishing fine-grained pixel-level associations between consecutive video frames.
Learning Propagation for Arbitrarily-structured Data
We experiment with semantic segmentation networks, where we use our propagation module to jointly train on different data -- images, superpixels and point clouds.
Few-Shot Viewpoint Estimation
Through extensive experimentation on the ObjectNet3D and Pascal3D+ benchmark datasets, we demonstrate that our framework, which we call MetaView, significantly outperforms fine-tuning the state-of-the-art models with few examples, and that the specific architectural innovations of our method are crucial to achieving good performance.
Few-Shot Adaptive Gaze Estimation
Inter-personal anatomical differences limit the accuracy of person-independent gaze estimation networks.
Ranked #1 on
Gaze Estimation
on MPII Gaze
(using extra training data)
Light-weight Head Pose Invariant Gaze Tracking
Unconstrained remote gaze tracking using off-the-shelf cameras is a challenging problem.
Switchable Temporal Propagation Network
Our approach is based on a temporal propagation network (TPN), which models the transition-related affinity between a pair of frames in a purely data-driven manner.
Learning Affinity via Spatial Propagation Network
Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be the output from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix.
Learning Affinity via Spatial Propagation Networks
Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be the output from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix.
Learning to Segment Instances in Videos with Spatial Propagation Network
In addition, we apply a filter on the refined score map that aims to recognize the best connected region using spatial and temporal consistencies in the video.
Dynamic Facial Analysis: From Bayesian Filtering to Recurrent Neural Network
We are inspired by the fact that the computation performed in an RNN bears resemblance to Bayesian filters, which have been used for tracking in many previous methods for facial analysis from videos.
Ranked #1 on
Head Pose Estimation
on BIWI
(MAE (trained with BIWI data) metric, using extra
training data)
