Search Results for author:
Found 93 papers, 51 papers with code
LoFTR: Detector-Free Local Feature Matching with Transformers
We present a novel method for local image feature matching.
CoDeF: Content Deformation Fields for Temporally Consistent Video Processing
We present the content deformation field CoDeF as a new type of video representation, which consists of a canonical content field aggregating the static contents in the entire video and a temporal deformation field recording the transformations from the canonical image (i. e., rendered from the canonical content field) to each individual frame along the time axis. Given a target video, these two fields are jointly optimized to reconstruct it through a carefully tailored rendering pipeline. We advisedly introduce some regularizations into the optimization process, urging the canonical content field to inherit semantics (e. g., the object shape) from the video. With such a design, CoDeF naturally supports lifting image algorithms for video processing, in the sense that one can apply an image algorithm to the canonical image and effortlessly propagate the outcomes to the entire video with the aid of the temporal deformation field. We experimentally show that CoDeF is able to lift image-to-image translation to video-to-video translation and lift keypoint detection to keypoint tracking without any training. More importantly, thanks to our lifting strategy that deploys the algorithms on only one image, we achieve superior cross-frame consistency in processed videos compared to existing video-to-video translation approaches, and even manage to track non-rigid objects like water and smog. Project page can be found at https://qiuyu96. github. io/CoDeF/.
Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views
This paper addresses the problem of 3D pose estimation for multiple people in a few calibrated camera views.
Ranked #12 on
3D Multi-Person Pose Estimation
on Campus
Motion Capture from Internet Videos
Therefore, we propose to capture human motion by jointly analyzing these Internet videos instead of using single videos separately.
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans
To this end, we propose Neural Body, a new human body representation which assumes that the learned neural representations at different frames share the same set of latent codes anchored to a deformable mesh, so that the observations across frames can be naturally integrated.
Reconstructing 3D Human Pose by Watching Humans in the Mirror
In this paper, we introduce the new task of reconstructing 3D human pose from a single image in which we can see the person and the person's image through a mirror.
NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video.
Deep Snake for Real-Time Instance Segmentation
Based on deep snake, we develop a two-stage pipeline for instance segmentation: initial contour proposal and contour deformation, which can handle errors in object localization.
Ranked #2 on
Semantic Contour Prediction
on Sbd val
OnePose: One-Shot Object Pose Estimation without CAD Models
We propose a new method named OnePose for object pose estimation.
PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation
We further create a Truncation LINEMOD dataset to validate the robustness of our approach against truncation.
Ranked #2 on
6D Pose Estimation using RGB
on YCB-Video
(Mean AUC metric)
Neural 3D Reconstruction in the Wild
We are witnessing an explosion of neural implicit representations in computer vision and graphics.
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies
Moreover, the learned blend weight fields can be combined with input skeletal motions to generate new deformation fields to animate the human model.
Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos
Some recent works have proposed to decompose a non-rigidly deforming scene into a canonical neural radiance field and a set of deformation fields that map observation-space points to the canonical space, thereby enabling them to learn the dynamic scene from images.
Neural 3D Scene Reconstruction with the Manhattan-world Assumption
Based on the Manhattan-world assumption, planar constraints are employed to regularize the geometry in floor and wall regions predicted by a 2D semantic segmentation network.
EasyVolcap: Accelerating Neural Volumetric Video Research
Volumetric video is a technology that digitally records dynamic events such as artistic performances, sporting events, and remote conversations.
Neural Rays for Occlusion-aware Image-based Rendering
On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction.
Detector-Free Structure from Motion
We propose a new detector-free SfM framework to draw benefits from the recent success of detector-free matchers to avoid the early determination of keypoints, while solving the multi-view inconsistency issue of detector-free matchers.
Coherent Reconstruction of Multiple Humans from a Single Image
Our goal is to train a single network that learns to avoid these problems and generate a coherent 3D reconstruction of all the humans in the scene.
Ranked #2 on
3D Human Reconstruction
on AGORA
PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos
We present PlanarRecon -- a novel framework for globally coherent detection and reconstruction of 3D planes from a posed monocular video.
Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed
Furthermore, we find spatial variance exists in LoFTR's fine correlation module, which is adverse to matching accuracy.
SMAP: Single-Shot Multi-Person Absolute 3D Pose Estimation
Recovering multi-person 3D poses with absolute scales from a single RGB image is a challenging problem due to the inherent depth and scale ambiguity from a single view.
Ranked #11 on
3D Multi-Person Pose Estimation (absolute)
on MuPoTS-3D
Disp R-CNN: Stereo 3D Object Detection via Shape Prior Guided Instance Disparity Estimation
In this paper, we propose a novel system named Disp R-CNN for 3D object detection from stereo images.
3D Object Detection From Stereo Images
Disparity Estimation
+2
Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation
In this work, we present a novel 3D-to-2D label transfer method, Panoptic NeRF, which aims for obtaining per-pixel 2D semantic and instance labels from easy-to-obtain coarse 3D bounding primitives.
TensoIR: Tensorial Inverse Rendering
We propose TensoIR, a novel inverse rendering approach based on tensor factorization and neural fields.
PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenes
Moreover, PanopticNeRF-360 enables omnidirectional rendering of high-fidelity, multi-view and spatiotemporally consistent appearance, semantic and instance labels.
GIFT: Learning Transformation-Invariant Dense Visual Descriptors via Group CNNs
Instead of feature pooling, we use group convolutions to exploit underlying structures of the extracted features on the group, resulting in descriptors that are both discriminative and provably invariant to the group of transformations.
Learning Feature Descriptors using Camera Pose Supervision
Recent research on learned visual descriptors has shown promising improvements in correspondence estimation, a key component of many 3D vision tasks.
Modeling Indirect Illumination for Inverse Rendering
The key insight is that indirect illumination can be conveniently derived from the neural radiance field learned from input images instead of being estimated jointly with direct illumination and materials.
Extreme Relative Pose Estimation for RGB-D Scans via Scene Completion
In particular, instead of only performing scene completion from each individual scan, our approach alternates between relative pose estimation and scene completion.
Learning Neural Volumetric Representations of Dynamic Humans in Minutes
In this paper, we propose a novel method for learning neural volumetric videos of dynamic humans from sparse view videos in minutes with competitive visual quality.
Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation
To enhance the quality of synthesized gestures, we develop a contrastive learning strategy based on audio-text alignment for better audio representations.
Ranked #3 on
Gesture Generation
on TED Gesture Dataset
Neural Scene Chronology
Specifically, we represent the scene as a space-time radiance field with a per-image illumination embedding, where temporally-varying scene changes are encoded using a set of learned step functions.
Ordinal Depth Supervision for 3D Human Pose Estimation
This information can be acquired by human annotators for a wide range of images and poses.
Ranked #1 on
Monocular 3D Human Pose Estimation
on Human3.6M
(Use Video Sequence metric)
6-DoF Object Pose from Semantic Keypoints
This paper presents a novel approach to estimating the continuous six degree of freedom (6-DoF) pose (3D translation and rotation) of an object from a single RGB image.
Ranked #1 on
Keypoint Detection
on Pascal3D+
VS-Net: Voting with Segmentation for Visual Localization
To address this problem, we propose a novel visual localization framework that establishes 2D-to-3D correspondences between the query image and the 3D map with a series of learnable scene-specific landmarks.
Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose
This paper addresses the challenge of 3D human pose estimation from a single color image.
Ranked #16 on
3D Human Pose Estimation
on HumanEva-I
Path-Invariant Map Networks
Optimizing a network of maps among a collection of objects/domains (or map synchronization) is a central problem across computer vision and many other relevant fields.
MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior
Recovering 3D full-body human pose is a challenging problem with many applications.
Polar Transformer Networks
The result is a network invariant to translation and equivariant to both rotation and scale.
Multi-Image Semantic Matching by Mining Consistent Features
This work proposes a multi-image matching method to estimate semantic correspondences across multiple images.
Learning Transformation Synchronization
Our approach alternates between transformation synchronization using weighted relative transformations and predicting new weights of the input relative transformations using a neural network.
SIRe-IR: Inverse Rendering for BRDF Reconstruction with Shadow and Illumination Removal in High-Illuminance Scenes
Implicit neural representation has opened up new possibilities for inverse rendering.
Reconstructing Close Human Interactions from Multiple Views
This paper addresses the challenging task of reconstructing the poses of multiple individuals engaged in close interactions, captured by multiple calibrated cameras.
Visual Sound Localization in the Wild by Cross-Modal Interference Erasing
Specifically, we observe that the previous practice of learning only a single audio representation is insufficient due to the additive nature of audio signals.
Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video
Here, two cases are considered: (i) the image locations of the human joints are provided and (ii) the image locations of joints are unknown.
Ranked #38 on
Monocular 3D Human Pose Estimation
on Human3.6M
Hybrid Convolutional and Attention Network for Hyperspectral Image Denoising
To enhance the modeling of both global and local features, we have devised a convolution and attention fusion module aimed at capturing long-range dependencies and neighborhood spectral correlations.
Human Motion Capture Using a Drone
Current motion capture (MoCap) systems generally require markers and multiple calibrated cameras, which can be used only in constrained environments.
EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior
This limitation leads to the Janus problem, where multi-faced 3D models are generated under the guidance of such diffusion models.
Multi-Image Matching via Fast Alternating Minimization
In this paper we propose a global optimization-based approach to jointly matching a set of images.
Semantic keypoint-based pose estimation from single RGB frames
Empirically, we show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios even against a cluttered background.
LADDER: Latent Boundary-guided Adversarial Training
To achieve a better trade-off between standard accuracy and adversarial robustness, we propose a novel adversarial training framework called LAtent bounDary-guided aDvErsarial tRaining (LADDER) that adversarially trains DNN models on latent boundary-guided adversarial examples.
Learning to Estimate 3D Human Pose and Shape from a Single Color Image
The proposed approach outperforms previous baselines on this task and offers an attractive solution for direct prediction of 3D shape from a single color image.
Ranked #111 on
3D Human Pose Estimation
on Human3.6M
(PA-MPJPE metric)
Harvesting Multiple Views for Marker-less 3D Human Pose Annotations
In this paper, we present a geometry-driven approach to automatically collect annotations for human pose prediction tasks.
Ranked #28 on
Weakly-supervised 3D Human Pose Estimation
on Human3.6M
Sparse Representation for 3D Shape Estimation: A Convex Relaxation Approach
We investigate the problem of estimating the 3D shape of an object defined by a set of 3D landmarks, given their 2D correspondences in a single image.
Ranked #118 on
3D Human Pose Estimation
on Human3.6M
(PA-MPJPE metric)
3D Shape Estimation from 2D Landmarks: A Convex Relaxation Approach
We investigate the problem of estimating the 3D shape of an object, given a set of 2D landmarks in a single image.
Pose and Shape Estimation with Discriminatively Learned Parts
We introduce a new approach for estimating the 3D pose and the 3D shape of an object from a single image.
Low-Rank Modeling and Its Applications in Image Analysis
In this paper, we review the recent advance of low-rank modeling, the state-of-the-art algorithms, and related applications in image analysis.
Moving Object Detection by Detecting Contiguous Outliers in the Low-Rank Representation
To automate the analysis, object detection without a separate training phase becomes a critical task.
Active Contours with Group Similarity
In this paper, we propose to use the group similarity of object shapes in multiple images as a prior to aid segmentation, which can be interpreted as an unsupervised approach of shape prior modeling.
Single Image Pop-Up From Discriminatively Learned Parts
We introduce a new approach for estimating a fine grained 3D shape and continuous pose of an object from a single image.
Fast Multi-Image Matching via Density-Based Clustering
We consider the problem of finding consistent matches across multiple images.
Latent Adversarial Defence with Boundary-guided Generation
The proposed LAD method improves the robustness of a DNN model through adversarial training on generated adversarial examples.
Monocular Human Pose and Shape Reconstruction using Part Differentiable Rendering
In this paper, we introduce body part segmentation as critical supervision.
Ranked #88 on
3D Human Pose Estimation
on Human3.6M
(PA-MPJPE metric)
Learning Hybrid Representations for Automatic 3D Vessel Centerline Extraction
The main idea is to use CNNs to learn local appearances of vessels in image crops while using another point-cloud network to learn the global geometry of vessels in the entire image.
Human-Understandable Decision Making for Visual Recognition
The widespread use of deep neural networks has achieved substantial success in many tasks.
Generative Transition Mechanism to Image-to-Image Translation via Encoded Transformation
To benefit the generalization ability of the translation model, we propose transition encoding to facilitate explicit regularization of these two {kinds} of consistencies on unseen transitions.
Edge but not Least: Cross-View Graph Pooling
Graph neural networks have emerged as a powerful model for graph representation learning to undertake graph-level prediction tasks.
You Don't Only Look Once: Constructing Spatial-Temporal Memory for Integrated 3D Object Detection and Tracking
In this work, we propose a novel system for integrated 3D object detection and tracking, which uses a dynamic object occupancy map and previous object states as spatial-temporal memory to assist object detection in future frames.
Efficient Neural Radiance Fields for Interactive Free-viewpoint Video
We propose a novel scene representation, called ENeRF, for the fast creation of interactive free-viewpoint videos.
Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation
Neural Radiance Fields (NeRF) have emerged as a potent paradigm for representing scenes and synthesizing photo-realistic images.
QuickPose: Real-time Multi-view Multi-person Pose Estimation in Crowded Scenes
The key challenge of this problem is to efficiently match 2D observations across multiple views.
Ranked #2 on
3D Multi-Person Pose Estimation
on Shelf
Reconstructing Hand-Held Objects from Monocular Video
The key idea is that the hand motion naturally provides multiple views of the object and the motion can be reliably estimated by a hand pose tracker.
Ponder: Point Cloud Pre-training via Neural Rendering
We propose a novel approach to self-supervised learning of point cloud representations by differentiable neural rendering.
OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Models
We propose a new method for object pose estimation without CAD models.
Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask
We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet.
Perceiving Unseen 3D Objects by Poking the Objects
We present a novel approach to interactive 3D object perception for robots.
Representing Volumetric Videos as Dynamic MLP Maps
This paper introduces a novel representation of volumetric videos for real-time view synthesis of dynamic scenes.
Long-term Visual Localization with Mobile Sensors
Despite the remarkable advances in image matching and pose estimation, image-based localization of a camera in a temporally-varying outdoor environment is still a challenging problem due to huge appearance disparity between query and reference images caused by illumination, seasonal and structural changes.
UTSGAN: Unseen Transition Suss GAN for Transition-Aware Image-to-image Translation
Furthermore, we propose the use of transition consistency, defined on the transition variable, to enable regularization of consistency on unobserved translations, which is omitted in previous works.
EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration
Hand-eye calibration is a critical task in robotics, as it directly affects the efficacy of critical operations such as manipulation and grasping.
AutoRecon: Automated 3D Object Discovery and Reconstruction
A fully automated object reconstruction pipeline is crucial for digital content creation.
Learning Human Mesh Recovery in 3D Scenes
We present a novel method for recovering the absolute pose and shape of a human in a pre-scanned scene given a single image.
Dyn-E: Local Appearance Editing of Dynamic Neural Radiance Fields
We extensively evaluate our approach on various scenes and show that our approach achieves spatially and temporally consistent editing results.
Relightable and Animatable Neural Avatar from Sparse-View Video
Based on the HDQ algorithm, we leverage sphere tracing to efficiently estimate the surface intersection and light visibility.
Deep Active Contours for Real-time 6-DoF Object Tracking
Specifically, given an initial pose, we project the object model to the image plane to obtain the initial contour and use a lightweight network to predict how the contour should move to match the true object boundary, which provides the gradients to optimize the object pose.
Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models
This paper presents a novel approach to generating the 3D motion of a human interacting with a target object, with a focus on solving the challenge of synthesizing long-range and diverse motions, which could not be fulfilled by existing auto-regressive models or path planning-based methods.
Im4D: High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes
This paper aims to tackle the challenge of dynamic view synthesis from multi-view videos.
4K4D: Real-Time 4D View Synthesis at 4K Resolution
Experiments show that our representation can be rendered at over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the ENeRF-Outdoor dataset at 4K resolution using an RTX 4090 GPU, which is 30x faster than previous methods and achieves the state-of-the-art rendering quality.
SAM-guided Graph Cut for 3D Instance Segmentation
Experimental results on the ScanNet, ScanNet++ and KITTI-360 datasets demonstrate that our method achieves robust segmentation performance and can generalize across different types of scenes.
Street Gaussians for Modeling Dynamic Urban Scenes
We introduce Street Gaussians, a new explicit scene representation that tackles all these limitations.
AniDress: Animatable Loose-Dressed Avatar from Sparse Views Using Garment Rigging Model
In this paper, we introduce AniDress, a novel method for generating animatable human avatars in loose clothes using very sparse multi-view videos (4-8 in our setting).
SpatialTracker: Tracking Any 2D Pixels in 3D Space
Recovering dense and long-range pixel motion in videos is a challenging problem.
IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
This paper aims to recover object materials from posed images captured under an unknown static lighting condition.
