Search Results for author:
Found 75 papers, 44 papers with code
Relightable Full-Body Gaussian Codec Avatars
This allows us to learn diffuse radiance transfer in a local coordinate frame, which disentangles the local radiance transfer from the articulation of the body.
SplatFormer: Point Transformer for Robust 3D Gaussian Splatting
To our knowledge, this is the first successful application of point transformers directly on 3DGS sets, surpassing the limitations of previous multi-scene training methods, which could handle only a restricted number of input views during inference.
DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control
Additionally, the learned motion primitive space allows for precise spatial motion control, which we formulate either as a latent noise optimization problem or as a Markov decision process addressed through reinforcement learning.
RISE-SDF: a Relightable Information-Shared Signed Distance Field for Glossy Object Inverse Rendering
Our experiments demonstrate that our algorithm achieves state-of-the-art performance in inverse rendering and relighting, with particularly strong results in the reconstruction of highly reflective objects.
SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction
Digitizing 3D static scenes and 4D dynamic events from multi-view images has long been a challenge in computer vision and graphics.
Improving 2D Feature Representations by 3D-Aware Fine-Tuning
In this work, we show that fine-tuning on 3D-aware data improves the quality of emerging semantic features.
Physics-Informed Learning of Characteristic Trajectories for Smoke Reconstruction
We delve into the physics-informed neural reconstruction of smoke and obstacles through sparse-view RGB videos, tackling challenges arising from limited observation of complex dynamics.
LaRa: Efficient Large-Baseline Radiance Fields
Radiance field methods have achieved photorealistic novel view synthesis and geometry reconstruction.
Degrees of Freedom Matter: Inferring Dynamics from Point Trajectories
By observing a set of point trajectories, we aim to learn an implicit motion field parameterized by a neural network to predict the movement of novel points within the same domain, without relying on any data-driven or scene-specific priors.
Portrait3D: Text-Guided High-Quality 3D Portrait Generation Using Pyramid Representation and GANs Prior
Existing neural rendering-based text-to-3D-portrait generation methods typically make use of human geometry prior and diffusion models to obtain guidance.
Creating a Digital Twin of Spinal Surgery: A Proof of Concept
Surgery digitalization is the process of creating a virtual replica of real-world surgery, also referred to as a surgical digital twin (SDT).
OpenSUN3D: 1st Workshop Challenge on Open-Vocabulary 3D Scene Understanding
This report provides an overview of the challenge hosted at the OpenSUN3D Workshop on Open-Vocabulary 3D Scene Understanding held in conjunction with ICCV 2023.
Is Continual Learning Ready for Real-world Challenges?
Our paper aims to initiate a paradigm shift, advocating for the adoption of continual learning methods through new experimental protocols that better emulate real-world conditions to facilitate breakthroughs in the field.
EgoGen: An Egocentric Synthetic Data Generator
To address this challenge, we introduce EgoGen, a new synthetic data generator that can produce accurate and rich ground-truth training data for egocentric perception tasks.
RoHM: Robust Human Motion Reconstruction via Diffusion
We apply RoHM to a variety of tasks -- from motion reconstruction and denoising to spatial and temporal infilling.
Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation
To the best of our knowledge, our proposed framework is the first diffusion model to enable the creation of fully 3D-consistent, animatable, and photorealistic human avatars from a single image of an unseen subject; extensive quantitative and qualitative evaluations demonstrate the advantages of our approach over existing state-of-the-art avatar creation models on both novel view and novel expression synthesis tasks.
Optimizing Diffusion Noise Can Serve As Universal Motion Priors
We propose Diffusion Noise Optimization (DNO), a new method that effectively leverages existing motion diffusion models as motion priors for a wide range of motion-related tasks.
3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting
In this paper, we use 3D Gaussian Splatting and learn a non-rigid deformation network to reconstruct animatable clothed human avatars that can be trained within 30 minutes and rendered at real-time frame rates (50+ FPS).
IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing
We present IntrinsicAvatar, a novel approach to recovering the intrinsic properties of clothed human avatars including geometry, albedo, material, and environment lighting from only monocular videos.
ResFields: Residual Neural Fields for Spatiotemporal Signals
Neural fields, a category of neural networks trained to represent high-frequency signals, have gained significant attention in recent years due to their impressive performance in modeling complex 3D data, such as signed distance (SDFs) or radiance fields (NeRFs), via a single multi-layer perceptron (MLP).
Synthesizing Diverse Human Motions in 3D Indoor Scenes
We present a novel method for populating 3D indoor scenes with virtual humans that can navigate in the environment and interact with objects in a realistic manner.
Guided Motion Diffusion for Controllable Human Motion Synthesis
Denoising diffusion models have shown great promise in human motion synthesis conditioned on natural language descriptions.
Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views
One of the biggest challenges of this task is severe body truncation due to close social distances in egocentric scenarios, which brings large pose ambiguities for unseen body parts.
Dynamic Point Fields
In this work, we present a dynamic point field model that combines the representational benefits of explicit point-based graphics with implicit deformation networks to allow efficient modeling of non-rigid 3D surfaces.
Factor Fields: A Unified Framework for Neural Fields and Beyond
Our experiments show that DiF leads to improvements in approximation quality, compactness, and training time when compared to previous fast reconstruction methods.
HARP: Personalized Hand Reconstruction from a Monocular RGB Video
We present HARP (HAnd Reconstruction and Personalization), a personalized hand avatar creation approach that takes a short monocular RGB video of a human hand as input and reconstructs a faithful hand avatar exhibiting a high-fidelity appearance and geometry.
3D Segmentation of Humans in Point Clouds with Synthetic Data
We address this challenge and propose a framework for generating training data of synthetic humans interacting with real 3D scenes.
ARAH: Animatable Volume Rendering of Articulated Human SDFs
We demonstrate that our proposed pipeline can generate clothed avatars with high-quality pose-dependent geometry and appearance from a sparse set of multi-view RGB videos.
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation
Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed geometric clustering techniques.
Ranked #2 on
3D Instance Segmentation
on STPLS3D
3D Instance Segmentation
3D Semantic Instance Segmentation
+2
Neural Point-based Shape Modeling of Humans in Challenging Clothing
Specifically, we extend point-based methods with a coarse stage, that replaces canonicalization with a learned pose-independent "coarse shape" that can capture the rough surface geometry of clothing like skirts.
Compositional Human-Scene Interaction Synthesis with Semantic Control
Furthermore, inspired by the compositional nature of interactions that humans can simultaneously interact with multiple objects, we define interaction semantics as the composition of varying numbers of atomic action-object pairs.
Accurate 3D Body Shape Regression using Metric and Semantic Attributes
Since paired data with images and 3D body shape are rare, we exploit two sources of information: (1) we collect internet images of diverse "fashion" models together with a small set of anthropometric measurements; (2) we collect linguistic shape attributes for a wide range of 3D body meshes and the model images.
Ranked #6 on
3D Human Shape Estimation
on SSP-3D
KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints
In this work, we investigate common issues with existing spatial encodings and propose a simple yet highly effective approach to modeling high-fidelity volumetric humans from sparse views.
Ranked #2 on
Generalizable Novel View Synthesis
on ZJU-MoCap
Context-Aware Sequence Alignment using 4D Skeletal Augmentation
Temporal alignment of fine-grained human actions in videos is important for numerous applications in computer vision, robotics, and mixed reality.
Interactive Object Segmentation in 3D Point Clouds
We propose an interactive approach for 3D instance segmentation, where users can iteratively collaborate with a deep learning model to segment objects in a 3D point cloud directly.
Ranked #1 on
Interactive 3D Instance Segmentation -Trained on Scannet40 - Evaluated on Scannet40
on ScanNetV2
Image Segmentation
Interactive 3D Instance Segmentation -Trained on Scannet40 - Evaluated on Scannet40
+4
COAP: Compositional Articulated Occupancy of People
We present a novel neural implicit representation for articulated human bodies.
Human-Aware Object Placement for Visual Environment Reconstruction
In fact, we demonstrate that these human-scene interactions (HSIs) can be leveraged to improve the 3D reconstruction of a scene from a monocular RGB video.
SAGA: Stochastic Whole-Body Grasping with Contact
Given an initial pose and the generated whole-body grasping pose as the start and end of the motion respectively, we design a novel contact-aware generative motion infilling module to generate a diverse set of grasp-oriented motions.
The Wanderings of Odysseus in 3D Scenes
In our solution, we decompose the long-term motion into a time sequence of motion primitives.
EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices
Key to reasoning about interactions is to understand the body pose and motion of the interaction partner from the egocentric view.
A Skeleton-Driven Neural Occupancy Representation for Articulated Hands
We present Hand ArticuLated Occupancy (HALO), a novel representation of articulated hands that bridges the advantages of 3D keypoints and neural implicit surfaces and can be used in end-to-end trainable architectures.
The Power of Points for Modeling Humans in Clothing
The geometry feature can be optimized to fit a previously unseen scan of a person in clothing, enabling the scan to be reposed realistically.
Learning Motion Priors for 4D Human Body Capture in 3D Scenes
To prove the effectiveness of the proposed motion priors, we combine them into a novel pipeline for 4D human body capture in 3D scenes.
Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation
In natural conversation and interaction, our hands often overlap or are in contact with each other.
Ranked #7 on
3D Interacting Hand Pose Estimation
on InterHand2.6M
MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images
In contrast, we propose an approach that can quickly generate realistic clothed human avatars, represented as controllable neural SDFs, given only monocular depth images.
Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration
We combine PTF with multi-class occupancy networks, obtaining a novel learning-based framework that learns to simultaneously predict shape and per-point correspondences between the posed space and the canonical space for clothed human.
SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements
We demonstrate the efficacy of our surface representation by learning models of complex clothing from point clouds.
LEAP: Learning Articulated Occupancy of People
Substantial progress has been made on modeling rigid 3D objects using deep implicit representations.
On Self-Contact and Human Pose
Third, we develop a novel HPS optimization method, SMPLify-XMC, that includes contact constraints and uses the known 3DCP body pose during fitting to create near ground-truth poses for MTP images.
Ranked #82 on
3D Human Pose Estimation
on 3DPW
(MPJPE metric)
We are More than Our Joints: Predicting how 3D Bodies Move
We note that motion prediction methods accumulate errors over time, resulting in joints or markers that diverge from true human bodies.
MATE: Plugging in Model Awareness to Task Embedding for Meta Learning
Meta-learning improves generalization of machine learning models when faced with previously unseen tasks by leveraging experiences from different, yet related prior tasks.
4D Human Body Capture from Egocentric Video via 3D Scene Grounding
We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos.
PLACE: Proximity Learning of Articulation and Contact in 3D Environments
To synthesize realistic human-scene interactions, it is essential to effectively represent the physical contact and proximity between the body and the world.
Grasping Field: Learning Implicit Representations for Human Grasps
Specifically, our generative model is able to synthesize high-quality human grasps, given only on a 3D object point cloud.
ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation
We show that our dataset can significantly improve the robustness of gaze estimation methods across different head poses and gaze angles.
Ranked #1 on
Gaze Estimation
on ETH-XGaze
(using extra training data)
Perpetual Motion: Generating Unbounded Human Motion
To address this problem, we propose a model to generate non-deterministic, \textit{ever-changing}, perpetual human motion, in which the global trajectory and the body pose are cross-conditioned.
Generating 3D People in Scenes without People
However, this is a challenging task for a computer as solving it requires that (1) the generated human bodies to be semantically plausible within the 3D environment (e. g. people sitting on the sofa or cooking near the stove), and (2) the generated human-scene interaction to be physically feasible such that the human body and scene do not interpenetrate while, at the same time, body-scene contact supports physical interactions.
Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video
Motivated by this, we adopt intentional hand movement as a future representation and propose a novel deep network that jointly models and predicts the egocentric hand motion, interaction hotspots and future action.
Learning Multi-Human Optical Flow
Therefore, we develop a dataset of multi-human optical flow and train optical flow networks on this dataset.
Learning to Train with Synthetic Humans
Here we explore two variations of synthetic data for this challenging problem; a dataset with purely synthetic humans and a real dataset augmented with synthetic humans.
Learning to Dress 3D People in Generative Clothing
To our knowledge, this is the first generative model that directly dresses 3D human body meshes and generalizes to different poses.
Frontal Low-rank Random Tensors for Fine-grained Action Segmentation
In this paper, we propose an approach to representing high-order information for temporal action segmentation via a simple yet effective bilinear form.
End-to-end Learning for Graph Decomposition
The new optimization problem can be viewed as a Conditional Random Field (CRF) in which the random variables are associated with the binary edge labels of the initial graph and the hard constraints are introduced in the CRF as high-order potentials.
Local Temporal Bilinear Pooling for Fine-grained Action Parsing
Fine-grained temporal action parsing is important in many applications, such as daily activity understanding, human motion analysis, surgical robotics and others requiring subtle and precise operations in a long-term period.
Part-Aligned Bilinear Representations for Person Re-identification
We propose a novel network that learns a part-aligned representation for person re-identification.
Ranked #4 on
Person Re-Identification
on UAV-Human
Temporal Human Action Segmentation via Dynamic Clustering
We present an effective dynamic clustering algorithm for the task of temporal human action segmentation, which has comprehensive applications such as robotics, motion analysis, and patient monitoring.
Multiple People Tracking by Lifted Multicut and Person Re-Identification
This allows us to reward tracks that assign detections of similar appearance to the same person in a way that does not introduce implausible solutions.
Joint Graph Decomposition & Node Labeling: Problem, Algorithms, Applications
In order to find feasible solutions efficiently, we define two local search algorithms that converge monotonously to a local optimum, offering a feasible solution at any time.
Generating Descriptions with Grounded and Co-Referenced People
At training time, we first learn how to localize characters by relating their visual appearance to mentions in the descriptions via a semi-supervised approach.
ArtTrack: Articulated Multi-person Tracking in the Wild
In this paper we propose an approach for articulated tracking of multiple people in unconstrained videos.
Ranked #7 on
Keypoint Detection
on MPII Multi-Person
Joint Graph Decomposition and Node Labeling: Problem, Algorithms, Applications
In order to find feasible solutions efficiently, we define two local search algorithms that converge monotonously to a local optimum, offering a feasible solution at any time.
Multi-Person Tracking by Multicut and Deep Matching
In [1], we proposed a graph-based formulation that links and clusters person hypotheses over time by solving a minimum cost subgraph multicut problem.
A Multi-cut Formulation for Joint Segmentation and Tracking of Multiple Objects
Recently, Minimum Cost Multicut Formulations have been proposed and proven to be successful in both motion trajectory segmentation and multi-target tracking scenarios.
DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation
This paper considers the task of articulated human pose estimation of multiple people in real world images.
Ranked #2 on
Multi-Person Pose Estimation
on WAF
Subgraph Decomposition for Multi-Target Tracking
Tracking multiple targets in a video, based on a finite set of detection hypotheses, is a persistent problem in computer vision.
