no code implementations • CVPR 2024 • Yizhou Zhao, Tuanfeng Y. Wang, Bhiksha Raj, Min Xu, Jimei Yang, Chun-Hao Paul Huang
Specifically, we design Human-aware Metric SLAM to reconstruct metric-scale camera poses and scene point clouds using camera-frame HMR as a strong prior, addressing depth, scale, and dynamic ambiguities.
no code implementations • 22 Feb 2024 • Yixuan Ren, Yang Zhou, Jimei Yang, Jing Shi, Difan Liu, Feng Liu, Mingi Kwon, Abhinav Shrivastava
To address the challenge of one-shot video motion customization, we propose Customize-A-Video that models the motion from a single reference video and adapts it to new subjects and scenes with both spatial and temporal varieties.
no code implementations • 22 Jan 2024 • Zhenzhen Weng, Jingyuan Liu, Hao Tan, Zhan Xu, Yang Zhou, Serena Yeung-Levy, Jimei Yang
We present Human-LRM, a diffusion-guided feed-forward model that predicts the implicit field of a human from a single image.
no code implementations • 19 Jan 2024 • Boxiao Pan, Zhan Xu, Chun-Hao Paul Huang, Krishna Kumar Singh, Yang Zhou, Leonidas J. Guibas, Jimei Yang
Generating video background that tailors to foreground subject motion is an important problem for the movie industry and visual effects community.
no code implementations • CVPR 2024 • Mohammad Amin Shabani, Zhaowen Wang, Difan Liu, Nanxuan Zhao, Jimei Yang, Yasutaka Furukawa
This paper proposes an image-vector dual diffusion model for generative layout design.
1 code implementation • ICCV 2023 • Shaowei Liu, Yang Zhou, Jimei Yang, Saurabh Gupta, Shenlong Wang
This paper presents a novel object-centric contact representation ContactGen for hand-object interaction.
1 code implementation • CVPR 2023 • Sumith Kulal, Tim Brooks, Alex Aiken, Jiajun Wu, Jimei Yang, Jingwan Lu, Alexei A. Efros, Krishna Kumar Singh
Given a scene image with a marked region and an image of a person, we insert the person into the scene while respecting the scene affordances.
no code implementations • CVPR 2023 • Yasamin Jafarian, Tuanfeng Y. Wang, Duygu Ceylan, Jimei Yang, Nathan Carr, Yi Zhou, Hyun Soo Park
To edit human videos in a physically plausible way, a texture map must take into account not only the garment transformation induced by the body movements and clothes fitting, but also its 3D fine-grained surface geometry.
1 code implementation • 23 Aug 2022 • Chun-Han Yao, Jimei Yang, Duygu Ceylan, Yi Zhou, Yang Zhou, Ming-Hsuan Yang
An alternative approach is to estimate dense vertices of a predefined template body in the image space.
1 code implementation • 28 Jul 2022 • Zhouyingcheng Liao, Jimei Yang, Jun Saito, Gerard Pons-Moll, Yang Zhou
We present the first method that automatically transfers poses between stylized 3D characters without skeletal rigging.
no code implementations • CVPR 2022 • Yang Zhou, Jimei Yang, DIngzeyu Li, Jun Saito, Deepali Aneja, Evangelos Kalogerakis
We present a method that reenacts a high-quality video with gestures matching a target speech audio.
no code implementations • 14 May 2022 • Yunseok Jang, Ruben Villegas, Jimei Yang, Duygu Ceylan, Xin Sun, Honglak Lee
We test the effectiveness of our representation on the human image harmonization task by predicting shading that is coherent with a given background image.
no code implementations • 1 May 2022 • Zhe Wang, Jimei Yang, Charless Fowlkes
Our framework leverages the best of non-parametric and model-based methods and is also robust to partial occlusion.
Ranked #1 on
3D Absolute Human Pose Estimation
on Human3.6M
(PA-MPJPE metric)
no code implementations • CVPR 2022 • Jae Shin Yoon, Duygu Ceylan, Tuanfeng Y. Wang, Jingwan Lu, Jimei Yang, Zhixin Shu, Hyun Soo Park
Appearance of dressed humans undergoes a complex geometric transformation induced not only by the static pose but also by its dynamics, i. e., there exists a number of cloth geometric configurations given a pose depending on the way it has moved.
no code implementations • ICCV 2021 • Ruben Villegas, Duygu Ceylan, Aaron Hertzmann, Jimei Yang, Jun Saito
Self-contacts, such as when hands touch each other or the torso or the head, are important attributes of human body language and dynamics, yet existing methods do not model or preserve these contacts.
no code implementations • 13 Sep 2021 • Badour AlBahar, Jingwan Lu, Jimei Yang, Zhixin Shu, Eli Shechtman, Jia-Bin Huang
We present an algorithm for re-rendering a person from a single image under arbitrary poses.
no code implementations • ICCV 2021 • Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, Michael Black
A long-standing goal in computer vision is to capture, model, and realistically synthesize human behavior.
no code implementations • 15 Jul 2021 • Manuel Lagunas, Xin Sun, Jimei Yang, Ruben Villegas, Jianming Zhang, Zhixin Shu, Belen Masia, Diego Gutierrez
We present a single-image data-driven method to automatically relight images with full-body humans in them.
no code implementations • 7 Jun 2021 • Jiaman Li, Ruben Villegas, Duygu Ceylan, Jimei Yang, Zhengfei Kuang, Hao Li, Yajie Zhao
We demonstrate the effectiveness of our hierarchical motion variational autoencoder in a variety of tasks including video-based human pose estimation, motion completion from partial observations, and motion synthesis from sparse key-frames.
Ranked #4 on
Motion Synthesis
on LaFAN1
1 code implementation • ICCV 2021 • Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, Leonidas J. Guibas
We introduce HuMoR: a 3D Human Motion Model for Robust Estimation of temporal pose and shape.
no code implementations • 11 Sep 2020 • Jianan Li, Jimei Yang, Jianming Zhang, Chang Liu, Christina Wang, Tingfa Xu
In this paper, we introduce Attribute-conditioned Layout GAN to incorporate the attributes of design elements for graphic layout generation by forcing both the generator and the discriminator to meet attribute conditions.
1 code implementation • ECCV 2020 • Davis Rempe, Leonidas J. Guibas, Aaron Hertzmann, Bryan Russell, Ruben Villegas, Jimei Yang
Existing deep models predict 2D and 3D kinematic poses from video that are approximately accurate, but contain visible errors that violate physical constraints, such as feet penetrating the ground and bodies leaning at extreme angles.
1 code implementation • ECCV 2020 • Yu Zeng, Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, Huchuan Lu
To address this challenge, we propose an iterative inpainting method with a feedback mechanism.
Ranked #8 on
Image Inpainting
on Places2
no code implementations • 18 May 2020 • Yi Zhou, Jingwan Lu, Connelly Barnes, Jimei Yang, Sitao Xiang, Hao Li
We introduce a biomechanically constrained generative adversarial network that performs long-term inbetweening of human motions, conditioned on keyframe constraints.
4 code implementations • 12 Sep 2019 • Simon Niklaus, Long Mai, Jimei Yang, Feng Liu
According to this depth estimate, our framework then maps the input image to a point cloud and synthesizes the resulting video frames by rendering the point cloud from the corresponding camera positions.
Ranked #4 on
Depth Estimation
on NYU-Depth V2
no code implementations • ICCV 2019 • Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, Thomas Brox
We show that methods trained on our dataset consistently perform well when tested on other datasets.
Ranked #24 on
3D Hand Pose Estimation
on FreiHAND
(PA-F@5mm metric)
no code implementations • 20 Aug 2019 • Yu-Wei Chao, Jimei Yang, Weifeng Chen, Jia Deng
We experimentally demonstrate the strength of our approach over different non-hierarchical and hierarchical baselines.
Deep Reinforcement Learning
Hierarchical Reinforcement Learning
+4
1 code implementation • ICLR 2019 • Jianan Li, Tingfa Xu, Jianming Zhang, Aaron Hertzmann, Jimei Yang
Layouts are important for graphic design and scene generation.
2 code implementations • ICCV 2019 • Yulun Zhang, Chen Fang, Yilin Wang, Zhaowen Wang, Zhe Lin, Yun Fu, Jimei Yang
An assumption widely used in recent neural style transfer methods is that image styles can be described by global statics of deep features like Gram or covariance matrices.
1 code implementation • 21 Jan 2019 • Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, Tingfa Xu
Layout is important for graphic design and scene generation.
no code implementations • CVPR 2019 • Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, Jiebo Luo
We show that by such disentanglement, the contour completion model predicts reasonable contours of objects, and further substantially improves the performance of image inpainting.
5 code implementations • CVPR 2019 • Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, Hao Li
Thus, widely used representations such as quaternions and Euler angles are discontinuous and difficult for neural networks to learn.
no code implementations • 14 Oct 2018 • Tao Zhou, Chen Fang, Zhaowen Wang, Jimei Yang, Byungmoon Kim, Zhili Chen, Jonathan Brandt, Demetri Terzopoulos
Doodling is a useful and common intelligent skill that people can learn and master.
1 code implementation • ECCV 2018 • Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, Ming-Hsuan Yang
Existing video prediction methods mainly rely on observing multiple historical frames or focus on predicting the next one-frame.
30 code implementations • ICCV 2019 • Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas Huang
We present a generative image inpainting system to complete images with free-form mask and guidance.
Ranked #3 on
Image Inpainting
on Places2 val
1 code implementation • CVPR 2018 • Chen Liu, Jimei Yang, Duygu Ceylan, Ersin Yumer, Yasutaka Furukawa
The proposed end-to-end DNN learns to directly infer a set of plane parameters and corresponding plane segmentation masks from a single RGB image.
Ranked #2 on
Plane Instance Segmentation
on NYU Depth v2
1 code implementation • CVPR 2018 • Ruben Villegas, Jimei Yang, Duygu Ceylan, Honglak Lee
We propose a recurrent neural network architecture with a Forward Kinematics layer and cycle consistency based adversarial training objective for unsupervised motion retargetting.
2 code implementations • ECCV 2018 • Gül Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, Cordelia Schmid
Human shape estimation is an important task for video editing, animation and fashion industry.
Ranked #3 on
3D Human Pose Estimation
on Surreal
(using extra training data)
28 code implementations • CVPR 2018 • Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S. Huang
Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions.
1 code implementation • CVPR 2018 • Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, Tamara L. Berg
In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression.
Generalized Referring Expression Segmentation
Referring Expression
+1
no code implementations • NeurIPS 2017 • Xiaojie Jin, Huaxin Xiao, Xiaohui Shen, Jimei Yang, Zhe Lin, Yunpeng Chen, Zequn Jie, Jiashi Feng, Shuicheng Yan
The ability of predicting the future is important for intelligent systems, e. g. autonomous vehicles and robots to plan early and make decisions accordingly.
no code implementations • ICCV 2017 • Xin Li, Zequn Jie, Wei Wang, Changsong Liu, Jimei Yang, Xiaohui Shen, Zhe Lin, Qiang Chen, Shuicheng Yan, Jiashi Feng
Thus, they suffer from heterogeneous object scales caused by perspective projection of cameras on actual scenes and inevitably encounter parsing failures on distant objects as well as other boundary and recognition errors.
2 code implementations • ICCV 2017 • Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, Derek Hoiem
The success of various applications including robotics, digital content creation, and visualization demand a structured and abstract representation of the 3D world from limited sensor data.
no code implementations • ICCV 2017 • Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, Jyh-Ming Lien
We propose an end-to-end network architecture that replicates the forward image formation process to accomplish this task.
no code implementations • 2 Jul 2017 • Ning Xu, Brian Price, Scott Cohen, Jimei Yang, Thomas Huang
In this paper, we propose a novel segmentation approach that uses a rectangle as a soft constraint by transforming it into an Euclidean distance map.
1 code implementation • 25 Jun 2017 • Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, Honglak Lee
To the best of our knowledge, this is the first end-to-end trainable network architecture with motion and content separation to model the spatiotemporal dynamics for pixel-level future prediction in natural videos.
Ranked #1 on
Video Prediction
on KTH
(Cond metric)
15 code implementations • NeurIPS 2017 • Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, Ming-Hsuan Yang
The whitening and coloring transforms reflect a direct matching of feature covariance of the content image to a given style image, which shares similar spirits with the optimization of Gram matrix based cost in neural style transfer.
2 code implementations • CVPR 2017 • Yijun Li, Sifei Liu, Jimei Yang, Ming-Hsuan Yang
In this paper, we propose an effective face completion algorithm using a deep generative model.
2 code implementations • ICML 2017 • Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, Honglak Lee
To avoid inherent compounding errors in recursive pixel-level prediction, we propose to first estimate high-level structure in the input frames, then predict how that structure evolves in the future, and finally by observing a single frame from the past and the predicted high-level structure, we construct the future frames without having to observe any of the pixel-level predictions.
no code implementations • CVPR 2017 • Yu-Wei Chao, Jimei Yang, Brian Price, Scott Cohen, Jia Deng
This paper presents the first study on forecasting human dynamics from static images.
1 code implementation • ICCV 2017 • Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Alan Yuille
In this paper we are interested in the problem of image segmentation given natural language descriptions, i. e. referring expressions.
2 code implementations • CVPR 2017 • Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, Alexander C. Berg
Instead of taking a 'blank slate' approach, we first explicitly infer the parts of the geometry visible both in the input and novel views and then re-cast the remaining synthesis problem as image completion.
no code implementations • CVPR 2017 • Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, Ming-Hsuan Yang
Recent progresses on deep discriminative and generative modeling have shown promising results on texture synthesis.
2 code implementations • NeurIPS 2016 • Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, Honglak Lee
We demonstrate the ability of the model in generating 3D volume from a single 2D image with three sets of experiments: (1) learning from single-class objects; (2) learning from multi-class objects and (3) testing on novel object classes.
no code implementations • ICCV 2017 • Xiaojie Jin, Xin Li, Huaxin Xiao, Xiaohui Shen, Zhe Lin, Jimei Yang, Yunpeng Chen, Jian Dong, Luoqi Liu, Zequn Jie, Jiashi Feng, Shuicheng Yan
In this way, the network can effectively learn to capture video dynamics and temporal context, which are critical clues for video scene parsing, without requiring extra manual annotations.
no code implementations • CVPR 2016 • Jifeng Ning, Jimei Yang, Shaojie Jiang, Lei Zhang, Ming-Hsuan Yang
Structured support vector machine (SSVM) based methods has demonstrated encouraging performance in recent object tracking benchmarks.
3 code implementations • CVPR 2016 • Jimei Yang, Brian Price, Scott Cohen, Honglak Lee, Ming-Hsuan Yang
We develop a deep learning algorithm for contour detection with a fully convolutional encoder-decoder network.
3 code implementations • CVPR 2016 • Ning Xu, Brian Price, Scott Cohen, Jimei Yang, Thomas Huang
Interactive object selection is a very important research problem and has many applications.
Ranked #11 on
Interactive Segmentation
on SBD
no code implementations • NeurIPS 2015 • Jimei Yang, Scott Reed, Ming-Hsuan Yang, Honglak Lee
An important problem for both graphics and vision is to synthesize novel views of a 3D object from a single image.
1 code implementation • 2 Dec 2015 • Xinchen Yan, Jimei Yang, Kihyuk Sohn, Honglak Lee
This paper investigates a novel problem of generating images from visual attributes.
no code implementations • CVPR 2015 • Sifei Liu, Jimei Yang, Chang Huang, Ming-Hsuan Yang
This paper formulates face labeling as a conditional random field with unary and pairwise classifiers.
no code implementations • CVPR 2015 • Jimei Yang, Brian Price, Scott Cohen, Zhe Lin, Ming-Hsuan Yang
The transferred local shape masks constitute a patch-level segmentation solution space and we thus develop a novel cascade algorithm, PatchCut, for coarse-to-fine object segmentation.
no code implementations • CVPR 2014 • Jimei Yang, Brian Price, Scott Cohen, Ming-Hsuan Yang
This paper presents a scalable scene parsing algorithm based on image retrieval and superpixel matching.
no code implementations • CVPR 2014 • Jimei Yang, Simon Safar, Ming-Hsuan Yang
We present Max-Margin Boltzmann Machines (MMBMs) for object segmentation.