Search Results for author:
Found 57 papers, 17 papers with code
VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors
To the best of our knowledge, this is the first generative approach to edit object compositions in videos.
On Unifying Video Generation and Camera Pose Estimation
Inspired by the emergent 3D capabilities in image generators, we explore whether video generators similarly exhibit 3D awareness.
Free-viewpoint Human Animation with Pose-correlated Reference Selection
Diffusion-based human animation aims to animate a human character based on a source human image as well as driving signals such as a sequence of poses.
GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space
We observe that GAN- and diffusion-based generators have complementary qualities: GANs can be trained efficiently with 2D supervision to produce high-quality 3D objects but are hard to condition on text.
Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation
While recent foundational video generators produce visually rich output, they still struggle with appearance drift, where objects gradually degrade or change inconsistently across frames, breaking visual coherence.
FloAt: Flow Warping of Self-Attention for Clothing Animation Generation
The input to our model is a text prompt depicting the type of clothing and the texture of clothing like leopard, striped, or plain, and a sequence of normal maps that capture the underlying animation that we desire in the output.
HyperGAN-CLIP: A Unified Framework for Domain Adaptation, Image Synthesis and Manipulation
Towards this end, in this study, we present a novel framework that significantly extends the capabilities of a pre-trained StyleGAN by integrating CLIP space via hypernetworks.
Boosting Camera Motion Control for Video Diffusion Transformers
While U-Net-based models have shown promising results for camera control, transformer-based diffusion models (DiT)-the preferred architecture for large-scale video generation - suffer from severe degradation in camera motion accuracy.
Learned Single-Pass Multitasking Perceptual Graphics for Immersive Displays
Uniquely, a single inference step of our model supports different permutations of these perceptual tasks at different prompted rates (i. e., mildly, lightly), eliminating the need for daisy-chaining multiple models to get the desired perceptual effect.
SuperGaussian: Repurposing Video Models for 3D Super Resolution
We demonstrate that it is possible to directly repurpose existing (pretrained) video models for 3D super-resolution and thus sidestep the problem of the shortage of large repositories of high-quality 3D training models.
Neural Garment Dynamics via Manifold-Aware Transformers
Data driven and learning based solutions for modeling dynamic garments have significantly advanced, especially in the context of digital humans.
SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models
In addition to audio conditioned image generation, our method can also be utilized in conjuction with diffusion based editing methods to enable audio conditioned image editing.
MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignment
We present MatAtlas, a method for consistent text-guided 3D model texturing.
Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models
Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path.
BLiSS: Bootstrapped Linear Shape Space
Morphable models are fundamental to numerous human-centered processes as they offer a simple yet expressive shape space.
GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency
In contrast, we introduce GRIP, a learning-based method that takes, as input, the 3D motion of the body and the object, and synthesizes realistic motion for both hands before, during, and after object interaction.
CLIP-Guided StyleGAN Inversion for Text-Driven Real Image Editing
Existing approaches for editing images using language either resort to instance-level latent code optimization or map predefined text prompts to some editing directions in the latent space.
VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs
However, the applicability of such advancements to the video domain has been hindered by the difficulty of representing and controlling videos in the latent space of GANs.
Neural Image-based Avatars: Generalizable Radiance Fields for Human Avatar Modeling
We present a method that enables synthesizing novel views and novel poses of arbitrary human performers from sparse multi-view images.
Pix2Video: Video Editing using Image Diffusion
Our method works in two simple steps: first, we use a pre-trained structure-guided (e. g., depth) image diffusion model to perform text-guided edits on an anchor frame; then, in the key step, we progressively propagate the changes to the future frames via self-attention feature injection to adapt the core denoising step of the diffusion model.
Blowing in the Wind: CycleNet for Human Cinemagraphs from Still Images
We investigate the problem in the context of dressed humans under the wind.
Normal-guided Garment UV Prediction for Human Re-texturing
To edit human videos in a physically plausible way, a texture map must take into account not only the garment transformation induced by the body movements and clothes fitting, but also its 3D fine-grained surface geometry.
UMFuse: Unified Multi View Fusion for Human Editing applications
Numerous pose-guided human editing methods have been explored by the vision community due to their extensive practical applications.
VGFlow: Visibility guided Flow Network for Human Reposing
The task of human reposing involves generating a realistic image of a person standing in an arbitrary conceivable pose.
Motion Guided Deep Dynamic 3D Garments
Technically, we model garment dynamics, driven using the input character motion, by predicting per-frame local displacements in a canonical state of the garment that is enriched with frame-dependent skinning weights to bring the garment to the global space.
Learning Visibility for Robust Dense Human Body Estimation
An alternative approach is to estimate dense vertices of a predefined template body in the image space.
A Repulsive Force Unit for Garment Collision Handling in Neural Networks
Despite recent success, deep learning-based methods for predicting 3D garment deformation under body motion suffer from interpenetration problems between the garment and the body.
RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects
We test the effectiveness of our representation on the human image harmonization task by predicting shading that is coherent with a given background image.
Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans from a Single Camera
Appearance of dressed humans undergoes a complex geometric transformation induced not only by the static pose but also by its dynamics, i. e., there exists a number of cloth geometric configurations given a pose depending on the way it has moved.
Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis
Synthesizing dynamic appearances of humans in motion plays a central role in applications such as AR/VR and video editing.
Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering
To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture.
Ranked #3 on
Generalizable Novel View Synthesis
on ZJU-MoCap
Contact-Aware Retargeting of Skinned Motion
Self-contacts, such as when hands touch each other or the torso or the head, are important attributes of human body language and dynamics, yet existing methods do not model or preserve these contacts.
CPFN: Cascaded Primitive Fitting Networks for High-Resolution Point Clouds
We present Cascaded Primitive Fitting Networks (CPFN) that relies on an adaptive patch sampling network to assemble detection results of global and local primitive detection networks.
Stochastic Scene-Aware Motion Prediction
A long-standing goal in computer vision is to capture, model, and realistically synthesize human behavior.
Task-Generic Hierarchical Human Motion Prior using VAEs
We demonstrate the effectiveness of our hierarchical motion variational autoencoder in a variety of tasks including video-based human pose estimation, motion completion from partial observations, and motion synthesis from sparse key-frames.
Ranked #4 on
Motion Synthesis
on LaFAN1
A Deep Emulator for Secondary Motion of 3D Characters
Being a local method, our network is independent of the mesh topology and generalizes to arbitrarily shaped 3D character meshes at test time.
Dynamic Neural Garments
A vital task of the wider digital human effort is the creation of realistic garments on digital avatars, both in the form of characteristic fold patterns and wrinkles in static frames as well as richness of garment dynamics under avatars' motion.
Intuitive, Interactive Beard and Hair Synthesis with Generative Models
We present an interactive approach to synthesizing realistic variations in facial hair in images, ranging from subtle edits to existing hair to the addition of complex and challenging hair in images of clean-shaven subjects.
Learning Generative Models of Shape Handles
We present a generative model to synthesize 3D shapes as sets of handles -- lightweight proxies that approximate the original 3D shape -- for applications in interactive editing, shape parsing, and building compact 3D representations.
Unsupervised Learning of Intrinsic Structural Representation Points
The 3D structure points produced by our method encode the shape structure intrinsically and exhibit semantic consistency across all the shape instances with similar structures.
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images
We show that methods trained on our dataset consistently perform well when tested on other datasets.
Ranked #24 on
3D Hand Pose Estimation
on FreiHAND
(PA-F@5mm metric)
DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
Reconstructing 3D shapes from single-view images has been a long-standing research problem.
Ranked #1 on
Single-View 3D Reconstruction
on ShapeNetCore
3DN: 3D Deformation Network
Given such a source 3D model and a target which can be a 2D image, 3D model, or a point cloud acquired as a depth scan, we introduce 3DN, an end-to-end network that deforms the source model to resemble the target.
SwapNet: Garment Transfer in Single View Images
Garment transfer is a challenging task that requires (i) disentangling the features of the clothing from the body pose and shape and (ii) realistic synthesis of the garment texture on the new body.
Ranked #1 on
Virtual Try-on
on FashionIQ
(using extra training data)
Learning a Shared Shape Space for Multimodal Garment Design
Designing real and virtual garments is becoming extremely demanding with rapidly changing fashion trends and increasing need for synthesizing realistic dressed digital humans for various applications.
Graphics
iMapper: Interaction-guided Joint Scene and Human Motion Mapping from Monocular Videos
A long-standing challenge in scene analysis is the recovery of scene arrangements under moderate to heavy occlusion, directly from monocular video.
PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image
The proposed end-to-end DNN learns to directly infer a set of plane parameters and corresponding plane segmentation masks from a single RGB image.
Ranked #2 on
Plane Instance Segmentation
on NYU Depth v2
Neural Kinematic Networks for Unsupervised Motion Retargetting
We propose a recurrent neural network architecture with a Forward Kinematics layer and cycle consistency based adversarial training objective for unsupervised motion retargetting.
BodyNet: Volumetric Inference of 3D Human Body Shapes
Human shape estimation is an important task for video editing, animation and fashion industry.
Ranked #3 on
3D Human Pose Estimation
on Surreal
(using extra training data)
Learning Dense Facial Correspondences in Unconstrained Images
To train such a network, we generate a massive dataset of synthetic faces with dense labels using renderings of a morphable face model with variations in pose, expressions, lighting, and occlusions.
3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks
The success of various applications including robotics, digital content creation, and visualization demand a structured and abstract representation of the 3D world from limited sensor data.
Material Editing Using a Physically Based Rendering Network
We propose an end-to-end network architecture that replicates the forward image formation process to accomplish this task.
Learning Local Shape Descriptors from Part Correspondences With Multi-view Convolutional Networks
We present a new local descriptor for 3D shapes, directly applicable to a wide range of shape analysis problems such as point correspondences, semantic segmentation, affordance prediction, and shape-to-scan matching.
Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
Instead of taking a 'blank slate' approach, we first explicitly infer the parts of the geometry visible both in the input and novel views and then re-cast the remaining synthesis problem as image completion.
Symmetry-aware Depth Estimation using Deep Neural Networks
Due to the abundance of 2D product images from the Internet, developing efficient and scalable algorithms to recover the missing depth information is central to many applications.
Capturing Dynamic Textured Surfaces of Moving Targets
We present an end-to-end system for reconstructing complete watertight and textured models of moving subjects such as clothed humans and animals, using only three or four handheld sensors.
Dense Human Body Correspondences Using Convolutional Networks
We propose a deep learning approach for finding dense correspondences between 3D scans of people.
