Search Results for author:
Found 100 papers, 26 papers with code
3D Fluid Flow Reconstruction Using Compact Light Field PIV
In this paper, we present a PIV solution that uses a compact lenslet-based light field camera to track dense particles floating in the fluid and reconstruct the 3D fluid flow.
IKOL: Inverse kinematics optimization layer for 3D human pose and shape estimation via Gauss-Newton differentiation
This paper presents an inverse kinematic optimization layer (IKOL) for 3D human pose and shape estimation that leverages the strength of both optimization- and regression-based methods within an end-to-end framework.
NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions
We construct a dense multi-view dome to acquire a complex human object interaction dataset, named HODome, that consists of $\sim$75M frames on 10 subjects interacting with 23 objects.
Relightable Neural Human Assets from Multi-view Gradient Illuminations
Human modeling and relighting are two fundamental problems in computer vision and graphics, where high-quality datasets can largely facilitate related research.
Executing your Commands via Motion Diffusion in Latent Space
We study a challenging task, conditional human motion generation, which produces plausible human motion sequences according to various conditional inputs, such as action classes or textual descriptors.
Ranked #1 on
Motion Synthesis
on KIT Motion-Language
Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDAR
Motivated by this, we propose a monocular camera and single LiDAR-based method for 3D multi-person pose estimation in large-scale scenes, which is easy to deploy and insensitive to light.
LiCamGait: Gait Recognition in the Wild by Using LiDAR and Camera Multi-modal Visual Sensors
LiDAR can capture accurate depth information in large-scale scenarios without the effect of light conditions, and the captured point cloud contains gait-related 3D geometric properties and dynamic motion characteristics.
Joint Rigid Motion Correction and Sparse-View CT via Self-Calibrating Neural Field
NeRF-based SVCT methods represent the desired CT image as a continuous function of spatial coordinates and train a Multi-Layer Perceptron (MLP) to learn the function by minimizing loss on the SV sinogram.
Human Performance Modeling and Rendering via Neural Animated Mesh
In this paper, we present a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos.
SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator
Named after the fossils of one of the oldest known human ancestors, our LUCY dataset contains high-quality Computed Tomography (CT) scans of the complete human head before and after orthognathic surgeries, critical for evaluating surgery results.
Self-Supervised Coordinate Projection Network for Sparse-View Computed Tomography
Compared with recent related works that solve similar problems using implicit neural representation network (INR), our essential contribution is an effective and simple re-projection strategy that pushes the tomography image reconstruction quality over supervised deep learning CT reconstruction works.
Generative Deformable Radiance Fields for Disentangled Image Synthesis of Topology-Varying Objects
Experiments show that our method can successfully learn the generative model from unstructured monocular images and well disentangle the shape and appearance for objects (e. g., chairs) with large topological variance.
Mutual Adaptive Reasoning for Monocular 3D Multi-Person Pose Estimation
This method first uses 2. 5D pose and geometry information to infer camera-centric root depths in a forward pass, and then exploits the root depths to further improve representation learning of 2. 5D pose estimation in a backward pass.
NARRATE: A Normal Assisted Free-View Portrait Stylizer
We extend Total Relighting to fix this problem by unifying its multi-view input normal maps with the physical face model.
LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors
We propose a multi-sensor fusion method for capturing challenging 3D human motions with accurate consecutive local poses and global trajectories in large-scale scenarios, only using a single LiDAR and 4 IMUs.
PREF: Phasorial Embedding Fields for Compact Neural Representations
We present an efficient frequency-based neural representation termed PREF: a shallow MLP augmented with a phasor volume that covers significant border spectra than previous Fourier feature mapping or Positional Encoding.
LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR Point Clouds
Quantitative and qualitative experiments show that our method outperforms the techniques based only on RGB images.
HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions
Monocular 3D motion capture (mocap) is beneficial to many applications.
Ranked #1 on
3D Human Pose Estimation
on AIST++
TensoRF: Tensorial Radiance Fields
We demonstrate that applying traditional CP decomposition -- that factorizes tensors into rank-one components with compact vectors -- in our framework leads to improvements over vanilla NeRF.
HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR
We propose Human-centered 4D Scene Capture (HSC4D) to accurately and efficiently create a dynamic digital world, containing large-scale indoor-outdoor scenes, diverse human motions, and rich interactions between humans and environments.
NeReF: Neural Refractive Field for Fluid Surface Reconstruction and Implicit Representation
We present a novel neural refractive field(NeReF) to recover wavefront of transparent fluids by simultaneously estimating the surface position and normal of the fluid front.
NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions
In this paper, we propose NeuralHOFusion, a neural approach for volumetric human-object capture and rendering using sparse consumer RGBD sensors.
Disentangling Light Fields for Super-Resolution and Disparity Estimation
In this paper, we propose a generic mechanism to disentangle these coupled information for LF image processing.
Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time
In this paper, we present a novel Fourier PlenOctree (FPO) technique to tackle efficient neural modeling and real-time rendering of dynamic scenes captured under the free-view video (FVV) setting.
NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing
We further develop a hybrid neural-rasterization rendering framework to support consumer-level VR headsets so that the aforementioned volumetric video viewing and editing, for the first time, can be conducted immersively in virtual 3D space.
Artemis: Articulated Neural Pets with Appearance and Motion synthesis
Our ARTEMIS enables interactive motion control, real-time animation, and photo-realistic rendering of furry animals.
Video-driven Neural Physically-based Facial Asset for Production
In this paper, we present a new learning-based, video-driven approach for generating dynamic facial geometries with high-quality physically-based assets.
NIMBLE: A Non-rigid Hand Model with Bones and Muscles
Emerging Metaverse applications demand reliable, accurate, and photorealistic reproductions of human hands to perform sophisticated operations as if in the physical world.
Onsite Non-Line-of-Sight Imaging via Online Calibrations
There has been an increasing interest in deploying non-line-of-sight (NLOS) imaging systems for recovering objects behind an obstacle.
HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs
The raw HumanNeRF can already produce reasonable rendering on sparse video inputs of unseen subjects and camera settings.
An Arbitrary Scale Super-Resolution Approach for 3D MR Images via Implicit Neural Representation
In the ArSSR model, the reconstruction of HR images with different up-scaling rates is defined as learning a continuous implicit voxel function from the observed LR images.
ProcK: Machine Learning for Knowledge-Intensive Processes
We present a novel methodology to build powerful predictive process models.
Light Field-Based Underwater 3D Reconstruction Via Angular Resampling
Recovering 3D geometry of underwater scenes is challenging because of non-linear refraction of light at the water-air interface caused by the camera housing.
iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering
To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effects.
Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions
In this paper, we propose a neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of both human and objects under challenging interaction scenarios in arbitrary novel views, from only sparse RGB streams.
Dynamic Reconstruction
Human-Object Interaction Detection
+3
Neural Relighting and Expression Transfer On Video Portraits
We present a neural relighting and expression transfer technique to transfer the facial expressions from a source performer to a portrait video of a target performer while enabling dynamic relighting.
Few-shot Neural Human Performance Rendering from Sparse RGBD Videos
To fill this gap, in this paper we propose a few-shot neural human rendering approach (FNHR) from only sparse RGBD inputs, which exploits the temporal and spatial redundancy to generate photo-realistic free-view output of human activities.
IREM: High-Resolution Magnetic Resonance (MR) Image Reconstruction via Implicit Neural Representation
For collecting high-quality high-resolution (HR) MR image, we propose a novel image reconstruction network named IREM, which is trained on multiple low-resolution (LR) MR images and achieve an arbitrary up-sampling rate for HR image reconstruction.
PIANO: A Parametric Hand Bone Model from Magnetic Resonance Imaging
Hand modeling is critical for immersive VR/AR, action understanding, or human healthcare.
Editable Free-viewpoint Video Using a Layered Neural Representation
Such layered representation supports fully perception and realistic manipulation of the dynamic scene whilst still supporting a free viewing experience in a wide range.
SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos
In this paper, we propose SportsCap -- the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input.
MirrorNeRF: One-shot Neural Portrait Radiance Field from Multi-mirror Catadioptric Imaging
In this paper, we propose MirrorNeRF - a one-shot neural portrait free-viewpoint rendering approach using a catadioptric imaging system with multiple sphere mirrors and a single high-resolution digital camera, which is the first to combine neural radiance field with catadioptric imaging so as to enable one-shot photo-realistic human portrait reconstruction and rendering, in a low-cost and casual capture setting.
Convolutional Neural Opacity Radiance Fields
In this paper, we propose a novel scheme to generate opacity radiance fields with a convolutional neural renderer for fuzzy objects, which is the first to combine both explicit opacity supervision and convolutional mechanism into the neural radiance field framework so as to enable high-quality appearance and global consistent alpha mattes generation in arbitrary novel views.
Neural Video Portrait Relighting in Real-time via Consistency Modeling
In this paper, we propose a neural approach for real-time, high-quality and coherent video portrait relighting, which jointly models the semantic, temporal and lighting consistency using a new dynamic OLAT dataset.
MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis.
GNeRF: GAN-based Neural Radiance Field without Posed Camera
We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field (NeRF) reconstruction for the complex scenarios with unknown and even randomly initialized camera poses.
Non-line-of-Sight Imaging via Neural Transient Fields
We present a neural modeling framework for Non-Line-of-Sight (NLOS) imaging.
LGNN: A Context-aware Line Segment Detector
We present a novel real-time line segment detection scheme called Line Graph Neural Network (LGNN).
SofGAN: A Portrait Image Generator with Dynamic Styling
To address this issue, we propose a SofGAN image generator to decouple the latent space of portraits into two subspaces: a geometry space and a texture space.
AutoSweep: Recovering 3D Editable Objectsfrom a Single Photograph
We base our work on the assumption that most human-made objects are constituted by parts and these parts can be well represented by generalized primitives.
OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer
A deep neural network of multiple nonlinear layers forms a large function space, which can easily lead to overfitting when it encounters small-sample data.
Spatial-Angular Interaction for Light Field Image Super-Resolution
Specifically, spatial and angular features are first separately extracted from input LFs, and then repetitively interacted to progressively incorporate spatial and angular information.
A Neural Rendering Framework for Free-Viewpoint Relighting
We present a novel Relightable Neural Renderer (RNR) for simultaneous view synthesis and relighting using multi-view image inputs.
Deep Coarse-to-fine Dense Light Field Reconstruction with Flexible Sampling and Geometry-aware Fusion
Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF.
Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses
To the best of our knowledge, this is the first end-to-end deep learning method for reconstructing a high-resolution LF image with a hybrid input.
Learning Semantics-aware Distance Map with Semantics Layering Network for Amodal Instance Segmentation
Specifically, we first introduce a new representation, namely a semantics-aware distance map (sem-dist map), to serve as our target for amodal segmentation instead of the commonly used masks and heatmaps.
PIV-Based 3D Fluid Flow Reconstruction Using Light Field Camera
Particle Imaging Velocimetry (PIV) estimates the flow of fluid by analyzing the motion of injected particles.
Non-Lambertian Surface Shape and Reflectance Reconstruction Using Concentric Multi-Spectral Light Field
In this paper, we introduce a novel concentric multi-spectral light field (CMSLF) design that is able to recover the shape and reflectance of surfaces with arbitrary material in one shot.
3D Face Reconstruction Using Color Photometric Stereo with Uncalibrated Near Point Lights
We avoid the need for spatial constancy of albedo; instead, we use a new measure for albedo similarity that is based on the albedo norm profile.
TightCap: 3D Human Shape Capture with Clothing Tightness Field
In this paper, we present TightCap, a data-driven scheme to capture both the human shape and dressed garments accurately with only a single 3D human scan, which enables numerous applications such as virtual try-on, biometrics and body evaluation.
Generic Multiview Visual Tracking
A natural solution to these challenges is to use multiple cameras with multiview inputs, though existing systems are mostly limited to specific targets (e. g. human), static cameras, and/or camera calibration.
Photo-Realistic Facial Details Synthesis from Single Image
Our technique employs expression analysis for proxy face geometry generation and combines supervised and unsupervised learning for facial detail synthesis.
Hair Segmentation on Time-of-Flight RGBD Images
Robust segmentation of hair from portrait images remains challenging: hair does not conform to a uniform shape, style or even color; dark hair in particular lacks features.
Deep Surface Light Fields
A surface light field represents the radiance of rays originating from any points on the surface in any directions.
4D Human Body Correspondences from Panoramic Depth Maps
The availability of affordable 3D full body reconstruction systems has given rise to free-viewpoint video (FVV) of human shapes.
Saliency Detection in 360° Videos
Considering that the 360° videos are usually stored with equirectangular panorama, we propose to implement the spherical convolution on panorama by stretching and rotating the kernel based on the location of patch to be convolved.
Learning to Dodge A Bullet: Concyclic View Morphing via Deep Learning
The bullet-time effect, presented in feature film ``The Matrix", has been widely adopted in feature films and TV commercials to create an amazing stopping-time illusion.
A Generic Multi-Projection-Center Model and Calibration Method for Light Field Cameras
Based on the MPC model and projective transformation, we propose a calibration algorithm to verify our light field camera model.
Focus Manipulation Detection via Photometric Histogram Analysis
With the rise of misinformation spread via social media channels, enabled by the increasing automation and realism of image manipulation tools, image forensics is an increasingly relevant problem.
Automatic 3D Indoor Scene Modeling From Single Panorama
The recovered layout is then used to guide shape estimation of the remaining objects using their normal information.
Gaze Prediction in Dynamic 360° Immersive Videos
This paper explores gaze prediction in dynamic $360^circ$ immersive videos, emph{i. e.}, based on the history scan path and VR contents, we predict where a viewer will look at an upcoming time.
Semantic See-Through Rendering on Light Fields
We present a novel semantic light field (LF) refocusing technique that can achieve unprecedented see-through quality.
Robust 3D Human Motion Reconstruction Via Dynamic Template Construction
In multi-view human body capture systems, the recovered 3D geometry or even the acquired imagery data can be heavily corrupted due to occlusions, noise, limited field of- view, etc.
Sparse Photometric 3D Face Reconstruction Guided by Morphable Models
We present a novel 3D face reconstruction technique that leverages sparse photometric stereo (PS) and latest advances on face registration/modeling from a single image.
Deep Eyes: Binocular Depth-from-Focus on Focal Stack Pairs
We then construct three individual networks: a Focus-Net to extract depth from a single focal stack, a EDoF-Net to obtain the extended depth of field (EDoF) image from the focal stack, and a Stereo-Net to conduct stereo matching.
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database.
Personalized Saliency and its Prediction
Specifically, we propose to decompose a personalized saliency map (referred to as PSM) into a universal saliency map (referred to as USM) predictable by existing saliency detection models and a new discrepancy map across users that characterizes personalized saliency.
Catadioptric HyperSpectral Light Field Imaging
In this paper, we present a single camera hyperspectral light field imaging solution that we call Snapshot Plenoptic Imager (SPI).
Ray Space Features for Plenoptic Structure-From-Motion
In this paper, we explore using light fields captured by plenoptic cameras or camera arrays as inputs.
A Learning-based Framework for Hybrid Depth-from-Defocus and Stereo Matching
Depth from defocus (DfD) and stereo matching are two most studied passive depth sensing schemes.
Image Splicing Detection via Camera Response Function Analysis
Recent advances on image manipulation techniques have made image forgery detection increasingly more challenging.
Robust Guided Image Filtering
In this paper, we propose a general framework for Robust Guided Image Filtering (RGIF), which contains a data term and a smoothness term, to solve the two issues mentioned above.
Occlusion-Model Guided Anti-Occlusion Depth Estimation in Light Field
Occlusion is one of the most challenging problems in depth estimation.
Rotational Crossed-Slit Light Field
In this paper, we present a novel LF sampling scheme by exploiting a special non-centric camera called the crossed-slit or XSlit camera.
Depth Recovery From Light Field Using Focal Stack Symmetry
The other is a data consistency measure based on analysis-by-synthesis, i. e., the difference between the synthesized focal stack given the hypothesized depth map and that from the LF.
Scene-adaptive Coded Apertures Imaging
Firstly, we capture the scene with a pinhole and analyze the scene content to determine primary edge orientations.
Robust High Quality Image Guided Depth Upsampling
A popular solution is upsampling the obtained noisy low resolution depth map with the guidance of the companion high resolution color image.
Automatic Layer Separation using Light Field Imaging
We propose a novel approach that jointly removes reflection or translucent layer from a scene and estimates scene depth.
Resolving Scale Ambiguity Via XSlit Aspect Ratio Analysis
We first conduct a comprehensive analysis to characterize DDAR, infer object depth from its AR, and model recoverable depth range, sensitivity, and error.
A Weighted Sparse Coding Framework for Saliency Detection
In this paper, we present a unified saliency detection framework for handling heterogenous types of input data.
Ambient Occlusion via Compressive Visibility Estimation
This enables a sparsity-prior based solution for iteratively recovering the surface normal, the surface albedo, and the visibility function from a small number of images.
Light Field Stereo Matching Using Bilateral Statistics of Surface Cameras
Our bilateral consistency metric is used to indicate the probability of occlusions by analyzing the SCams.
Saliency Detection on Light Field
Existing saliency detection approaches use images as inputs and are sensitive to foreground/background similarities, complex background textures, and occlusions.
Curvilinear Structure Tracking by Low Rank Tensor Approximation with Model Propagation
Robust tracking of deformable object like catheter or vascular structures in X-ray images is an important technique used in image guided medical interventions for effective motion compensation and dynamic multi-modality image fusion.
Aliasing Detection and Reduction in Plenoptic Imaging
When using plenoptic camera for digital refocusing, angular undersampling can cause severe (angular) aliasing artifacts.
Image Pre-compensation: Balancing Contrast and Ringing
In particular, we show that linear tone mapping eliminates ringing but incurs severe contrast loss, while non-linear tone mapping functions such as Gamma curves slightly enhances contrast but introduces ringing.
Reconstructing Gas Flows Using Light-Path Approximation
By observing the LF-probe through the gas flow, we acquire a dense set of ray-ray correspondences and then reconstruct their light paths.
Fast Patch-Based Denoising Using Approximated Patch Geodesic Paths
PatchGPs treat image patches as nodes and patch differences as edge weights for computing the shortest (geodesic) paths.
Manhattan Scene Understanding via XSlit Imaging
Specifically, we prove that parallel 3D lines map to 2D curves in an XSlit image and they converge at an XSlit Vanishing Point (XVP).
