Search Results for author:
Found 84 papers, 21 papers with code
RobustFusion: Human Volumetric Capture with Data-driven Visual Cues using a RGBD Camera
To enable robust tracking, we embrace both the initial model and the various visual cues into a novel performance capture scheme with hybrid motion optimization and semantic volumetric fusion, which can successfully capture challenging human motions under the monocular setting without pre-scanned detailed template and owns the reinitialization ability to recover from tracking failures and the disappear-reoccur scenarios.
LLaVA-SLT: Visual Language Tuning for Sign Language Translation
We propose hierarchical visual encoder that learns a robust word-level intermediate representation that is compatible with LLM token embeddings.
CADSpotting: Robust Panoptic Symbol Spotting on Large-Scale CAD Drawings
We introduce CADSpotting, an efficient method for panoptic symbol spotting in large-scale architectural CAD drawings.
SMGDiff: Soccer Motion Generation using diffusion probabilistic models
Soccer is a globally renowned sport with significant applications in video games and VR/AR.
V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic Gaussians
However, current dynamic 3DGS methods, despite their high rendering quality, face challenges in streaming on mobile devices due to computational and bandwidth constraints.
Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos
In this paper, we present a novel Gaussian-based approach, dubbed \textit{DualGS}, for real-time and high-fidelity playback of complex human performance with excellent compression ratios.
HiSC4D: Human-centered interaction and 4D Scene Capture in Large-scale Space Using Wearable IMUs and LiDAR
To promote research of egocentric human interaction in large scenes and facilitate downstream tasks, we also present a dataset, containing 8 sequences in 4 large scenes (200 to 5, 000 $m^2$), providing 36k frames of accurate 4D human motions with SMPL annotations and dynamic scenes, 31k frames of cropped human point clouds, and scene mesh of the environment.
SCOPE: Sign Language Contextual Processing with Embedding from LLMs
Current methods in vision-based sign language recognition (SLR) and translation (SLT) struggle with dialogue scenes due to limited dataset diversity and the neglect of contextually relevant information.
HeadGAP: Few-Shot 3D Head Avatar via Generalizable Gaussian Priors
The prior learning phase leverages 3D head priors derived from a large-scale multi-view dynamic dataset, and the avatar creation phase applies these priors for few-shot personalization.
StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset
Modeling and capturing the 3D spatial arrangement of the human and the object is the key to perceiving 3D human-object interaction from monocular images.
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets
To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures.
Know in AdVance: Linear-Complexity Forecasting of Ad Campaign Performance with Evolving User Interest
We further condition SSM's global predictions on the accumulation of local results.
LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives
Large garages are ubiquitous yet intricate scenes that present unique challenges due to their monotonous colors, repetitive patterns, reflective surfaces, and transparent vehicle glass.
A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals
One of the biggest challenges to this task is the one-to-many mapping from sparse observations to dense full-body motions, which endowed inherent ambiguities.
HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environment
Humans naturally interact with both others and the surrounding multiple objects, engaging in various social activities.
RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method
Comprehensive capturing of human motions requires both accurate captures of complex poses and precise localization of the human within scenes.
Gaze-guided Hand-Object Interaction Synthesis: Dataset and Method
Gaze plays a crucial role in revealing human attention and intention, particularly in hand-object interaction scenarios, where it guides and synchronizes complex tasks that require precise coordination between the brain, hand, and object.
THOR: Text to Human-Object Interaction Diffusion via Relation Intervention
This paper addresses new methodologies to deal with the challenging task of generating dynamic Human-Object Interactions from textual descriptions (Text2HOI).
LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment
For human-centric large-scale scenes, fine-grained modeling for 3D human global pose and shape is significant for scene understanding and can benefit many real-world applications.
GaussianHair: Hair Modeling and Rendering with Light-aware Gaussians
Hairstyle reflects culture and ethnicity at first glance.
IMUSE: IMU-based Facial Expression Capture
In this paper, we propose IMUSE to fill the gap, a novel path for facial expression capture using purely IMU signals, significantly distant from previous visual solutions. The key design in our IMUSE is a trilogy.
DressCode: Autoregressively Sewing and Generating Garments from Text Guidance
We first introduce SewingGPT, a GPT-based architecture integrating cross-attention with text-conditioned embedding to generate sewing patterns with text guidance.
Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance
The synthesis of 3D facial animations from speech has garnered considerable attention.
HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment
Humans naturally interact with both others and the surrounding multiple objects engaging in various social activities.
HybridGait: A Benchmark for Spatial-Temporal Cloth-Changing Gait Recognition with Hybrid Explorations
Existing gait recognition benchmarks mostly include minor clothing variations in the laboratory environments, but lack persistent changes in appearance over time and space.
MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D Human Generation
Recent months have witnessed rapid progress in 3D generation based on diffusion models.
OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers
At the subsequent fine-tuning stage, we introduce motion ControlNet, which incorporates text prompts as conditioning information, through a trainable copy of the pre-trained model and the proposed novel Mixture-of-Controllers (MoC) block.
BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics
We further provide a strong baseline method, BOTH2Hands, for the novel task: generating vivid two-hand motions from both implicit body dynamics and explicit text prompts.
I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions
In this paper, we present I'm-HOI, a monocular scheme to faithfully capture the 3D motions of both the human and object in a novel setting: using a minimal amount of RGB camera and object-mounted Inertial Measurement Unit (IMU).
HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models
We further present a strong baseline method HandDiffuse for the controllable motion generation of interacting hands using various controllers.
HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting
Then, we utilize a 4D Gaussian optimization scheme with adaptive spatial-temporal regularizers to effectively balance the non-rigid prior and Gaussian updating.
VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams
In this paper, we introduce VideoRF, the first approach to enable real-time streaming and rendering of dynamic radiance fields on mobile platforms.
Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator
Text-to-video is a rapidly growing research area that aims to generate a semantic, identical, and temporal coherence sequence of frames that accurately align with the input text prompt.
InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions
Then, we propose a novel representation for motion input in our interaction diffusion model, which explicitly formulates the global relations between the two performers in the world frame.
Ranked #2 on
Motion Synthesis
on Inter-X
Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos
The success of the Neural Radiance Fields (NeRFs) for modeling and free-view rendering static objects has inspired numerous attempts on dynamic scenes.
Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Stream
We also provide an on-the-fly reconstruction scheme of the dynamic/static radiance fields via efficient motion-prior searching.
CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions
The core of this dataset is a blending optimization process, which corrects for the pose as it drifts and is affected by the magnetic conditions.
CryoFormer: Continuous Heterogeneous Cryo-EM Reconstruction using Transformer-based Neural Representations
Cryo-electron microscopy (cryo-EM) allows for the high-resolution reconstruction of 3D structures of proteins and other biomolecules.
SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments
We present SLOPER4D, a novel scene-aware dataset collected in large urban environments to facilitate the research of global human pose estimation (GHPE) with human-scene interaction in the wild.
NEPHELE: A Neural Platform for Highly Realistic Cloud Radiance Rendering
Using such a cloud platform compatible with neural rendering, we further showcase the capabilities of our cloud radiance rendering through a series of applications, ranging from cloud VR/AR rendering.
IKOL: Inverse kinematics optimization layer for 3D human pose and shape estimation via Gauss-Newton differentiation
This paper presents an inverse kinematic optimization layer (IKOL) for 3D human pose and shape estimation that leverages the strength of both optimization- and regression-based methods within an end-to-end framework.
Ranked #32 on
3D Human Pose Estimation
on 3DPW
NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions
We construct a dense multi-view dome to acquire a complex human object interaction dataset, named HODome, that consists of $\sim$75M frames on 10 subjects interacting with 23 objects.
Relightable Neural Human Assets from Multi-view Gradient Illuminations
UltraStage will be publicly available to the community to stimulate significant future developments in various human modeling and rendering tasks.
HumanGen: Generating Human Radiance Fields with Explicit Priors
With the aid of the anchor image, we adapt a 3D reconstructor for fine-grained details synthesis and propose a two-stage blending scheme to boost appearance generation.
Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDAR
Motivated by this, we propose a monocular camera and single LiDAR-based method for 3D multi-person pose estimation in large-scale scenes, which is easy to deploy and insensitive to light.
Gait Recognition in Large-scale Free Environment via Single LiDAR
Human gait recognition is crucial in multimedia, enabling identification through walking patterns without direct interaction, enhancing the integration across various media forms in real-world applications like smart homes, healthcare and non-intrusive security.
Learning Variational Motion Prior for Video-based Motion Capture
To improve the generalization capacity of prior space, we propose a transformer-based variational autoencoder pretrained over marker-based 3D mocap data, with a novel style-mapping block to boost the generation quality.
Human Performance Modeling and Rendering via Neural Animated Mesh
In this paper, we present a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos.
SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator
Named after the fossils of one of the oldest known human ancestors, our LUCY dataset contains high-quality Computed Tomography (CT) scans of the complete human head before and after orthognathic surgeries, critical for evaluating surgery results.
Mutual Adaptive Reasoning for Monocular 3D Multi-Person Pose Estimation
This method first uses 2. 5D pose and geometry information to infer camera-centric root depths in a forward pass, and then exploits the root depths to further improve representation learning of 2. 5D pose estimation in a backward pass.
NARRATE: A Normal Assisted Free-View Portrait Stylizer
We extend Total Relighting to fix this problem by unifying its multi-view input normal maps with the physical face model.
LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors
We propose a multi-sensor fusion method for capturing challenging 3D human motions with accurate consecutive local poses and global trajectories in large-scale scenarios, only using single LiDAR and 4 IMUs, which are set up conveniently and worn lightly.
STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenes
In addition, considering the property of sparse global distribution and density-varying local distribution of pedestrians, we further propose a novel method, Density-aware Hierarchical heatmap Aggregation (DHA), to enhance pedestrian perception in crowded scenes.
LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR Point Clouds
Quantitative and qualitative experiments show that our method outperforms the techniques based only on RGB images.
Ranked #3 on
3D Human Pose Estimation
on SLOPER4D
(using extra training data)
HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR
We propose Human-centered 4D Scene Capture (HSC4D) to accurately and efficiently create a dynamic digital world, containing large-scale indoor-outdoor scenes, diverse human motions, and rich interactions between humans and environments.
HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions
Monocular 3D motion capture (mocap) is beneficial to many applications.
Ranked #1 on
Pose Estimation
on 3DPW
NeReF: Neural Refractive Field for Fluid Surface Reconstruction and Implicit Representation
We present a novel neural refractive field(NeReF) to recover wavefront of transparent fluids by simultaneously estimating the surface position and normal of the fluid front.
NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions
In this paper, we propose NeuralHOFusion, a neural approach for volumetric human-object capture and rendering using sparse consumer RGBD sensors.
Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time
In this paper, we present a novel Fourier PlenOctree (FPO) technique to tackle efficient neural modeling and real-time rendering of dynamic scenes captured under the free-view video (FVV) setting.
NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing
We further develop a hybrid neural-rasterization rendering framework to support consumer-level VR headsets so that the aforementioned volumetric video viewing and editing, for the first time, can be conducted immersively in virtual 3D space.
Video-driven Neural Physically-based Facial Asset for Production
In this paper, we present a new learning-based, video-driven approach for generating dynamic facial geometries with high-quality physically-based assets.
Artemis: Articulated Neural Pets with Appearance and Motion synthesis
Our ARTEMIS enables interactive motion control, real-time animation, and photo-realistic rendering of furry animals.
NIMBLE: A Non-rigid Hand Model with Bones and Muscles
Emerging Metaverse applications demand reliable, accurate, and photorealistic reproductions of human hands to perform sophisticated operations as if in the physical world.
HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs
The raw HumanNeRF can already produce reasonable rendering on sparse video inputs of unseen subjects and camera settings.
Towards Controllable and Photorealistic Region-wise Image Manipulation
Adaptive and flexible image editing is a desirable function of modern generative models.
iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering
To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effects.
Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions
In this paper, we propose a neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of both human and objects under challenging interaction scenarios in arbitrary novel views, from only sparse RGB streams.
Neural Relighting and Expression Transfer On Video Portraits
We present a neural relighting and expression transfer technique to transfer the facial expressions from a source performer to a portrait video of a target performer while enabling dynamic relighting.
Few-shot Neural Human Performance Rendering from Sparse RGBD Videos
To fill this gap, in this paper we propose a few-shot neural human rendering approach (FNHR) from only sparse RGBD inputs, which exploits the temporal and spatial redundancy to generate photo-realistic free-view output of human activities.
IREM: High-Resolution Magnetic Resonance (MR) Image Reconstruction via Implicit Neural Representation
For collecting high-quality high-resolution (HR) MR image, we propose a novel image reconstruction network named IREM, which is trained on multiple low-resolution (LR) MR images and achieve an arbitrary up-sampling rate for HR image reconstruction.
PIANO: A Parametric Hand Bone Model from Magnetic Resonance Imaging
Hand modeling is critical for immersive VR/AR, action understanding, or human healthcare.
Editable Free-viewpoint Video Using a Layered Neural Representation
Such layered representation supports fully perception and realistic manipulation of the dynamic scene whilst still supporting a free viewing experience in a wide range.
RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream
To fill this gap, in this paper, we propose RobustFusion, a robust volumetric performance reconstruction system for human-object interaction scenarios using only a single RGBD sensor, which combines various data-driven visual and interaction cues to handle the complex interaction patterns and severe occlusions.
SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos
In this paper, we propose SportsCap -- the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input.
MirrorNeRF: One-shot Neural Portrait Radiance Field from Multi-mirror Catadioptric Imaging
In this paper, we propose MirrorNeRF - a one-shot neural portrait free-viewpoint rendering approach using a catadioptric imaging system with multiple sphere mirrors and a single high-resolution digital camera, which is the first to combine neural radiance field with catadioptric imaging so as to enable one-shot photo-realistic human portrait reconstruction and rendering, in a low-cost and casual capture setting.
Convolutional Neural Opacity Radiance Fields
In this paper, we propose a novel scheme to generate opacity radiance fields with a convolutional neural renderer for fuzzy objects, which is the first to combine both explicit opacity supervision and convolutional mechanism into the neural radiance field framework so as to enable high-quality appearance and global consistent alpha mattes generation in arbitrary novel views.
Neural Video Portrait Relighting in Real-time via Consistency Modeling
In this paper, we propose a neural approach for real-time, high-quality and coherent video portrait relighting, which jointly models the semantic, temporal and lighting consistency using a new dynamic OLAT dataset.
GNeRF: GAN-based Neural Radiance Field without Posed Camera
We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field (NeRF) reconstruction for the complex scenarios with unknown and even randomly initialized camera poses.
NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras
4D reconstruction and rendering of human activities is critical for immersive VR/AR experience. Recent advances still fail to recover fine geometry and texture results with the level of detail present in the input images from sparse multi-view RGB cameras.
ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References
We propose a hybrid motion inference stage with a generation network, which utilizes a temporal encoder-decoder to extract the motion details from the pair-wise sparse-view reference, as well as a motion discriminator to utilize the unpaired marker-based references to extract specific challenging motion characteristics in a data-driven manner.
OccuSeg: Occupancy-aware 3D Instance Segmentation
3D instance segmentation, with a variety of applications in robotics and augmented reality, is in large demands these days.
Ranked #1 on
3D Instance Segmentation
on SceneNN
EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera
The high frame rate is a critical requirement for capturing fast human motions.
Beyond SIFT using Binary features for Loop Closure Detection
The proposed system originates from our previous work Multi-Index hashing for Loop closure Detection (MILD), which employs Multi-Index Hashing (MIH)~\cite{greene1994multi} for Approximate Nearest Neighbor (ANN) search of binary features.
FlyCap: Markerless Motion Capture Using Multiple Autonomous Flying Cameras
We propose a novel non-rigid surface registration method to track and fuse the depth of the three flying cameras for surface motion tracking of the moving target, and simultaneously calculate the pose of each flying camera.
