Search Results for author: Zhixin Shu

Found 28 papers, 5 papers with code

Rig3DGS: Creating Controllable Portraits from Casual Monocular Videos

no code implementations6 Feb 2024 Alfredo Rivero, ShahRukh Athar, Zhixin Shu, Dimitris Samaras

Using a set of control signals, such as head pose and expressions, we transform them to the 3D space with learned deformations to generate the desired rendering.

Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning

no code implementations21 Dec 2023 Desai Xie, Jiahao Li, Hao Tan, Xin Sun, Zhixin Shu, Yi Zhou, Sai Bi, Sören Pirk, Arie E. Kaufman

To this end, we introduce Carve3D, a RLFT method coupled with the Multi-view Reconstruction Consistency (MRC) metric, to improve the consistency of multi-view diffusion models.

Text to 3D

Relightful Harmonization: Lighting-aware Portrait Background Replacement

no code implementations11 Dec 2023 Mengwei Ren, Wei Xiong, Jae Shin Yoon, Zhixin Shu, Jianming Zhang, HyunJoon Jung, Guido Gerig, He Zhang

Portrait harmonization aims to composite a subject into a new background, adjusting its lighting and color to ensure harmony with the background scene.

Controllable Dynamic Appearance for Neural 3D Portraits

no code implementations20 Sep 2023 ShahRukh Athar, Zhixin Shu, Zexiang Xu, Fujun Luan, Sai Bi, Kalyan Sunkavalli, Dimitris Samaras

The surface normals prediction is guided using 3DMM normals that act as a coarse prior for the normals of the human head, where direct prediction of normals is hard due to rigid and non-rigid deformations induced by head-pose and facial expression changes.

Consistent Multimodal Generation via A Unified GAN Framework

no code implementations4 Jul 2023 Zhen Zhu, Yijun Li, Weijie Lyu, Krishna Kumar Singh, Zhixin Shu, Soeren Pirk, Derek Hoiem

We investigate how to generate multimodal image outputs, such as RGB, depth, and surface normals, with a single generative model.

multimodal generation

LightPainter: Interactive Portrait Relighting with Freehand Scribble

no code implementations CVPR 2023 Yiqun Mei, He Zhang, Xuaner Zhang, Jianming Zhang, Zhixin Shu, Yilin Wang, Zijun Wei, Shi Yan, HyunJoon Jung, Vishal M. Patel

Recent portrait relighting methods have achieved realistic results of portrait lighting effects given a desired lighting representation such as an environment map.

In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing

no code implementations9 Feb 2023 Yiran Xu, Zhixin Shu, Cameron Smith, Seoung Wug Oh, Jia-Bin Huang

3D-aware GANs offer new capabilities for view synthesis while preserving the editing functionalities of their 2D counterparts.

3D-FM GAN: Towards 3D-Controllable Face Manipulation

no code implementations24 Aug 2022 Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, S. Y. Kung

While concatenating GAN inversion and a 3D-aware, noise-to-image GAN is a straight-forward solution, it is inefficient and may lead to noticeable drop in editing quality.

RigNeRF: Fully Controllable Neural 3D Portraits

no code implementations CVPR 2022 ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, Zhixin Shu

In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video.

Face Model Neural Rendering +1

Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans from a Single Camera

no code implementations CVPR 2022 Jae Shin Yoon, Duygu Ceylan, Tuanfeng Y. Wang, Jingwan Lu, Jimei Yang, Zhixin Shu, Hyun Soo Park

Appearance of dressed humans undergoes a complex geometric transformation induced not only by the static pose but also by its dynamics, i. e., there exists a number of cloth geometric configurations given a pose depending on the way it has moved.

Point-NeRF: Point-based Neural Radiance Fields

1 code implementation CVPR 2022 Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, Ulrich Neumann

Point-NeRF combines the advantages of these two approaches by using neural 3D point clouds, with associated neural features, to model a radiance field.

3D Reconstruction Neural Rendering

FLAME-in-NeRF: Neural control of Radiance Fields for Free View Face Animation

no code implementations29 Sep 2021 ShahRukh Athar, Zhixin Shu, Dimitris Samaras

In this work, we design a system that enables 1) novel view synthesis for portrait video, of both the human subject and the scene they are in and 2) explicit control of the facial expressions through a low-dimensional expression representation.

Neural Rendering Novel View Synthesis

Learning Surface Parameterization for Document Image Unwarping

no code implementations29 Sep 2021 Sagnik Das, Ke Ma, Zhixin Shu, Dimitris Samaras

We also demonstrate the usefulness of our system by applying it to document texture editing.

3D Scene Reconstruction

FLAME-in-NeRF : Neural control of Radiance Fields for Free View Face Animation

no code implementations10 Aug 2021 ShahRukh Athar, Zhixin Shu, Dimitris Samaras

In this work, we design a system that enables both novel view synthesis for portrait video, including the human subject and the scene background, and explicit control of the facial expressions through a low-dimensional expression representation.

Face Model Neural Rendering +1

Single-image Full-body Human Relighting

no code implementations15 Jul 2021 Manuel Lagunas, Xin Sun, Jimei Yang, Ruben Villegas, Jianming Zhang, Zhixin Shu, Belen Masia, Diego Gutierrez

We present a single-image data-driven method to automatically relight images with full-body humans in them.

Image Reconstruction

Content-Aware GAN Compression

1 code implementation CVPR 2021 Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Federico Perazzi, S. Y. Kung

We then propose a novel content-aware method to guide the processes of both pruning and distillation.

Image Generation Image Manipulation +1

Learning Clusterable Visual Features for Zero-Shot Recognition

no code implementations7 Oct 2020 Jingyi Xu, Zhixin Shu, Dimitris Samaras

However, some testing data are considered "hard" as they lie close to the decision boundaries and are prone to misclassification, leading to performance degradation for ZSL.

Classification Few-Shot Learning +2

Self-supervised Deformation Modeling for Facial Expression Editing

no code implementations2 Nov 2019 ShahRukh Athar, Zhixin Shu, Dimitris Samaras

In the "motion-editing" step, we explicitly model facial movement through image deformation, warping the image into the desired expression.

Disentanglement Facial Editing +2

Latent Space Optimal Transport for Generative Models

no code implementations16 Sep 2018 Huidong Liu, Yang Guo, Na lei, Zhixin Shu, Shing-Tung Yau, Dimitris Samaras, Xianfeng GU

Experimental results on an eight-Gaussian dataset show that the proposed OT can handle multi-cluster distributions.

DocUNet: Document Image Unwarping via a Stacked U-Net

1 code implementation CVPR 2018 Ke Ma, Zhixin Shu, Xue Bai, Jue Wang, Dimitris Samaras

The network is trained on this dataset with various data augmentations to improve its generalization ability.

Ranked #4 on SSIM on DocUNet (using extra training data)

Local Distortion MS-SSIM +1

An Adversarial Neuro-Tensorial Approach For Learning Disentangled Representations

no code implementations28 Nov 2017 Mengjiao Wang, Zhixin Shu, Shiyang Cheng, Yannis Panagakis, Dimitris Samaras, Stefanos Zafeiriou

Several factors contribute to the appearance of an object in a visual scene, including pose, illumination, and deformation, among others.

3D Face Reconstruction

Improving Heterogeneous Face Recognition with Conditional Adversarial Networks

no code implementations8 Sep 2017 Wuming Zhang, Zhixin Shu, Dimitris Samaras, Liming Chen

Heterogeneous face recognition between color image and depth image is a much desired capacity for real world applications where shape information is looked upon as merely involved in gallery.

Face Recognition Heterogeneous Face Recognition +1

Neural Face Editing with Intrinsic Image Disentangling

2 code implementations CVPR 2017 Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, Dimitris Samaras

Traditional face editing methods often require a number of sophisticated and task specific algorithms to be applied one after the other --- a process that is tedious, fragile, and computationally intensive.

Facial Editing Generative Adversarial Network

Cannot find the paper you are looking for? You can Submit a new open access paper.