Search Results for author: Zhengyi Luo

Found 24 papers, 7 papers with code

ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills

1 code implementation3 Feb 2025 Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbab, Chaoyi Pan, Zeji Yi, Guannan Qu, Kris Kitani, Jessica Hodgins, Linxi "Jim" Fan, Yuke Zhu, Changliu Liu, Guanya Shi

In the second stage, we deploy the policies in the real world and collect real-world data to train a delta (residual) action model that compensates for the dynamics mismatch.

Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions

no code implementations27 Oct 2024 Rawal Khirodkar, Jyun-Ting Song, Jinkun Cao, Zhengyi Luo, Kris Kitani

Understanding how humans interact with each other is key to building realistic multi-human virtual reality systems.

3D Pose Estimation Human Detection

CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control

1 code implementation4 Oct 2024 Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue Bin Peng, Amit H. Bermano, Michiel Van de Panne

The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment.

Motion Generation Reinforcement Learning (RL)

SkillMimic: Learning Reusable Basketball Skills from Demonstrations

no code implementations12 Aug 2024 Yinhuai Wang, Qihan Zhao, Runyi Yu, Ailing Zeng, Jing Lin, Zhengyi Luo, Hok Wai Tsui, Jiwen Yu, Xiu Li, Qifeng Chen, Jian Zhang, Lei Zhang, Ping Tan

SkillMimic employs a unified configuration to learn diverse skills from human-ball motion datasets, with skill diversity and generalization improving as the dataset grows.

SMPLOlympics: Sports Environments for Physically Simulated Humanoids

no code implementations28 Jun 2024 Zhengyi Luo, Jiashun Wang, Kangni Liu, Haotian Zhang, Chen Tessler, Jingbo Wang, Ye Yuan, Jinkun Cao, Zihui Lin, Fengyi Wang, Jessica Hodgins, Kris Kitani

We present SMPLOlympics, a collection of physically simulated environments that allow humanoids to compete in a variety of Olympic sports.

OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning

no code implementations13 Jun 2024 Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, Guanya Shi

We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy.

PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios

no code implementations CVPR 2024 Jingbo Wang, Zhengyi Luo, Ye Yuan, Yixuan Li, Bo Dai

We address the challenge of content diversity and controllability in pedestrian simulation for driving scenarios.

Diversity

Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation

no code implementations7 Mar 2024 Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, Guanya Shi

We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB camera.

Reinforcement Learning (RL)

PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction

no code implementations7 Dec 2023 Yinhuai Wang, Jing Lin, Ailing Zeng, Zhengyi Luo, Jian Zhang, Lei Zhang

To make up for the lack of dynamic HOI scenarios in this area, we introduce the BallPlay dataset that contains eight whole-body basketball skills.

Human-Object Interaction Detection Object

Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives

2 code implementations CVPR 2024 Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei HUANG, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram, Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Yan, Xitong Yang, Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelaez, Gedas Bertasius, David Crandall, Dima Damen, Jakob Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C. V. Jawahar, Richard Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, Michael Wray

We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.

Video Understanding

Perpetual Humanoid Control for Real-time Simulated Avatars

no code implementations ICCV 2023 Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, Weipeng Xu

We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior in the presence of noisy input (e. g. pose estimates from video or generated from language) and unexpected falls.

Humanoid Control

Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion

no code implementations CVPR 2023 Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, Or Litany

We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals.

Collision Avoidance

Learning Human Dynamics in Autonomous Driving Scenarios

no code implementations ICCV 2023 Jingbo Wang, Ye Yuan, Zhengyi Luo, Kevin Xie, Dahua Lin, Umar Iqbal, Sanja Fidler, Sameh Khamis

In this work, we propose a holistic framework for learning physically plausible human dynamics from real driving scenarios, narrowing the gap between real and simulated human behavior in safety-critical applications.

Autonomous Driving Human Dynamics

From Universal Humanoid Control to Automatic Physically Valid Character Creation

no code implementations18 Jun 2022 Zhengyi Luo, Ye Yuan, Kris M. Kitani

Second, we use a design-and-control framework to optimize a humanoid's physical attributes to find body designs that can better imitate the pre-specified human motion sequence(s).

Humanoid Control valid

Embodied Scene-aware Human Pose Estimation

no code implementations18 Jun 2022 Zhengyi Luo, Shun Iwase, Ye Yuan, Kris Kitani

Since 2D third-person observations are coupled with the camera pose, we propose to disentangle the camera pose and use a multi-step projection gradient defined in the global coordinate frame as the movement cue for our embodied agent.

3D Human Pose Estimation Causal Inference +1

Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design

1 code implementation ICLR 2022 Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, Kris Kitani

Specifically, we learn a conditional policy that, in an episode, first applies a sequence of transform actions to modify an agent's skeletal structure and joint attributes, and then applies control actions under the new design.

Decision Making Policy Gradient Methods

Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation

1 code implementation NeurIPS 2021 Zhengyi Luo, Ryo Hachiuma, Ye Yuan, Kris Kitani

By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model.

Egocentric Pose Estimation Human-Object Interaction Detection +2

Kinematics-Guided Reinforcement Learning for Object-Aware 3D Ego-Pose Estimation

no code implementations10 Nov 2020 Zhengyi Luo, Ryo Hachiuma, Ye Yuan, Shun Iwase, Kris M. Kitani

We propose a method for incorporating object interaction and human body dynamics into the task of 3D ego-pose estimation using a head-mounted camera.

Human-Object Interaction Detection Object +4

3D Human Motion Estimation via Motion Compression and Refinement

2 code implementations9 Aug 2020 Zhengyi Luo, S. Alireza Golestaneh, Kris M. Kitani

Experiments show that our method produces both smooth and accurate 3D human pose and motion estimates.

Ranked #16 on 3D Human Pose Estimation on 3DPW (Acceleration Error metric, using extra training data)

3D Human Pose Estimation

Learning Shape Representations for Clothing Variations in Person Re-Identification

no code implementations16 Mar 2020 Yu-Jhe Li, Zhengyi Luo, Xinshuo Weng, Kris M. Kitani

To tackle the re-ID problem in the context of clothing changes, we propose a novel representation learning model which is able to generate a body shape feature representation without being affected by clothing color or patterns.

Disentanglement Person Re-Identification

Cross-Domain 3D Equivariant Image Embeddings

1 code implementation6 Dec 2018 Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, Ameesh Makadia

This embedding encodes images with 3D shape properties and is equivariant to 3D rotations of the observed object.

3D Shape Classification Novel View Synthesis +2

Cloud Chaser: Real Time Deep Learning Computer Vision on Low Computing Power Devices

no code implementations2 Oct 2018 Zhengyi Luo, Austin Small, Liam Dugan, Stephen Lane

Internet of Things(IoT) devices, mobile phones, and robotic systems are often denied the power of deep learning algorithms due to their limited computing power.

Cloud Computing Deep Learning

Cannot find the paper you are looking for? You can Submit a new open access paper.