Search Results for author: Otmar Hilliges

Found 74 papers, 34 papers with code

Human from Blur: Human Pose Tracking from Blurry Images

no code implementations30 Mar 2023 Yiming Zhao, Denys Rozumnyi, Jie Song, Otmar Hilliges, Marc Pollefeys, Martin R. Oswald

The key idea is to tackle the inverse problem of image deblurring by modeling the forward problem with a 3D human model, a texture map, and a sequence of poses to describe human motion.

Learning Human-to-Robot Handovers from Point Clouds

no code implementations30 Mar 2023 Sammy Christen, Wei Yang, Claudia Pérez-D'Arpino, Otmar Hilliges, Dieter Fox, Yu-Wei Chao

We propose the first framework to learn control policies for vision-based human-to-robot handovers, a critical task for human-robot interaction.

Hi4D: 4D Instance Segmentation of Close Human Interaction

no code implementations27 Mar 2023 Yifei Yin, Chen Guo, Manuel Kaufmann, Juan Jose Zarate, Jie Song, Otmar Hilliges

We propose Hi4D, a method and dataset for the automatic analysis of physically close human-human interaction under prolonged contact.

Instance Segmentation Semantic Segmentation

Efficient Learning of High Level Plans from Play

no code implementations16 Mar 2023 Núria Armengol Urpí, Marco Bagatella, Otmar Hilliges, Georg Martius, Stelian Coros

Real-world robotic manipulation tasks remain an elusive challenge, since they involve both fine-grained environment interaction, as well as the ability to plan for long-horizon goals.

Motion Planning Reinforcement Learning (RL)

X-Avatar: Expressive Human Avatars

1 code implementation8 Mar 2023 Kaiyue Shen, Chen Guo, Manuel Kaufmann, Juan Jose Zarate, Julien Valentin, Jie Song, Otmar Hilliges

Our method models bodies, hands, facial expressions and appearance in a holistic fashion and can be learned from either full 3D scans or RGB-D data.

3D Human Reconstruction

Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition

no code implementations22 Feb 2023 Chen Guo, Tianjian Jiang, Xu Chen, Jie Song, Otmar Hilliges

Specifically, we define a temporally consistent human representation in canonical space and formulate a global optimization over the background model, the canonical human shape and texture, and per-frame human pose parameters.

Surface Reconstruction

InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds

no code implementations20 Dec 2022 Tianjian Jiang, Xu Chen, Jie Song, Otmar Hilliges

To achieve this efficiency we propose a carefully designed and engineered system, that leverages emerging acceleration structures for neural fields, in combination with an efficient empty space-skipping strategy for dynamic scenes.

HARP: Personalized Hand Reconstruction from a Monocular RGB Video

no code implementations19 Dec 2022 Korrawe Karunratanakul, Sergey Prokudin, Otmar Hilliges, Siyu Tang

We present HARP (HAnd Reconstruction and Personalization), a personalized hand avatar creation approach that takes a short monocular RGB video of a human hand as input and reconstructs a faithful hand avatar exhibiting a high-fidelity appearance and geometry.

3D Hand Pose Estimation

PointAvatar: Deformable Point-based Head Avatars from Videos

1 code implementation16 Dec 2022 Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, Otmar Hilliges

The ability to create realistic, animatable and relightable head avatars from casual video sequences would open up wide ranging applications in communication and entertainment.

HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics

no code implementations14 Dec 2022 Artur Grigorev, Bernhard Thomaszewski, Michael J. Black, Otmar Hilliges

We propose a method that leverages graph neural networks, multi-level message passing, and unsupervised training to enable real-time prediction of realistic clothing dynamics.

Fast-SNARF: A Fast Deformer for Articulated Neural Fields

1 code implementation28 Nov 2022 Xu Chen, Tianjian Jiang, Jie Song, Max Rietmann, Andreas Geiger, Michael J. Black, Otmar Hilliges

A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space.

3D Reconstruction Novel View Synthesis

Robust Real-Time Tracking of Axis-Symmetric Magnets via Neural Networks

no code implementations14 Nov 2022 Mengfan Wu, Thomas Langerak, Otmar Hilliges, Juan Zarate

We propose to overcome these limitations by using neural networks to infer the marker's position and orientation.

Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors

no code implementations6 Sep 2022 Xi Wang, Gen Li, Yen-Ling Kuo, Muhammed Kocabas, Emre Aksan, Otmar Hilliges

We further qualitatively evaluate the effectiveness of our method on real images and demonstrate its generalizability towards interaction types and object categories.

Human-Object Interaction Detection

TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning

no code implementations1 Sep 2022 Andrea Ziani, Zicong Fan, Muhammed Kocabas, Sammy Christen, Otmar Hilliges

We introduce TempCLR, a new time-coherent contrastive learning approach for the structured regression task of 3D hand reconstruction.

Contrastive Learning Hand Pose Estimation

EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes

no code implementations16 Jun 2022 Gengyan Li, Abhimitra Meka, Franziska Müller, Marcel C. Bühler, Otmar Hilliges, Thabo Beeler

The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling skin and eye appearance from environmental illumination such that it may be rendered under novel lighting conditions, and 3) capturing eyeball motion and the deformation of the surrounding skin to enable re-gazing.

SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learning

no code implementations26 May 2022 Marco Bagatella, Sammy Christen, Otmar Hilliges

Several methods, such as behavioral priors, are able to leverage offline data in order to efficiently accelerate reinforcement learning on complex tasks.

Continuous Control Efficient Exploration +2

Articulated Objects in Free-form Hand Interaction

no code implementations28 Apr 2022 Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, Otmar Hilliges

Consequently, we introduce ARCTIC - the first dataset of free-form interactions of hands and articulated objects.

3D Reconstruction

LiP-Flow: Learning Inference-time Priors for Codec Avatars via Normalizing Flows in Latent Space

no code implementations15 Mar 2022 Emre Aksan, Shugao Ma, Akin Caliskan, Stanislav Pidhorskyi, Alexander Richard, Shih-En Wei, Jason Saragih, Otmar Hilliges

To mitigate this asymmetry, we introduce a prior model that is conditioned on the runtime inputs and tie this prior space to the 3D face model via a normalizing flow in the latent space.

Face Model

gDNA: Towards Generative Detailed Neural Avatars

no code implementations CVPR 2022 Xu Chen, Tianjian Jiang, Jie Song, Jinlong Yang, Michael J. Black, Andreas Geiger, Otmar Hilliges

Furthermore, we show that our method can be used on the task of fitting human models to raw scans, outperforming the previous state-of-the-art.

SAGA: Stochastic Whole-Body Grasping with Contact

no code implementations19 Dec 2021 Yan Wu, Jiahao Wang, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, Siyu Tang

Given an initial pose and the generated whole-body grasping pose as the start and end of the motion respectively, we design a novel contact-aware generative motion infilling module to generate a diverse set of grasp-oriented motions.

I M Avatar: Implicit Morphable Head Avatars from Videos

1 code implementation CVPR 2022 Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, Otmar Hilliges

Traditional 3D morphable face models (3DMMs) provide fine-grained control over expression but cannot easily capture geometric and appearance details.

D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions

1 code implementation CVPR 2022 Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, Otmar Hilliges

We introduce the dynamic grasp synthesis task: given an object with a known 6D pose and a grasp reference, our goal is to generate motions that move the object to a target 6D pose.

Motion Synthesis

Human Performance Capture from Monocular Video in the Wild

1 code implementation29 Nov 2021 Chen Guo, Xu Chen, Jie Song, Otmar Hilliges

In this work, we propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input.

3D Human Shape Estimation Autonomous Driving

Render In-between: Motion Guided Video Synthesis for Action Interpolation

no code implementations1 Nov 2021 Hsuan-I Ho, Xu Chen, Jie Song, Otmar Hilliges

We propose to address these issues in a motion-guided frame-upsampling framework that is capable of producing realistic human motion and appearance.

Neural Rendering

A Skeleton-Driven Neural Occupancy Representation for Articulated Hands

no code implementations23 Sep 2021 Korrawe Karunratanakul, Adrian Spurr, Zicong Fan, Otmar Hilliges, Siyu Tang

We present Hand ArticuLated Occupancy (HALO), a novel representation of articulated hands that bridges the advantages of 3D keypoints and neural implicit surfaces and can be used in end-to-end trainable architectures.

PeCLR: Self-Supervised 3D Hand Pose Estimation from monocular RGB via Equivariant Contrastive Learning

1 code implementation ICCV 2021 Adrian Spurr, Aneesh Dahiya, Xi Wang, Xucong Zhang, Otmar Hilliges

Encouraged by the success of contrastive learning on image classification tasks, we propose a new self-supervised method for the structured regression task of 3D hand pose estimation.

3D Hand Pose Estimation Contrastive Learning +3

Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation

no code implementations10 Jun 2021 Adrian Spurr, Pavlo Molchanov, Umar Iqbal, Jan Kautz, Otmar Hilliges

Hand pose estimation is difficult due to different environmental conditions, object- and self-occlusion as well as diversity in hand shape and appearance.

Hand Pose Estimation

PARE: Part Attention Regressor for 3D Human Body Estimation

1 code implementation ICCV 2021 Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, Michael J. Black

Despite significant progress, we show that state of the art 3D human pose and shape estimation methods remain sensitive to partial occlusion and can produce dramatically wrong predictions although much of the body is observable.

3D human pose and shape estimation 3D Multi-Person Pose Estimation

VariTex: Variational Neural Face Textures

1 code implementation ICCV 2021 Marcel C. Bühler, Abhimitra Meka, Gengyan Li, Thabo Beeler, Otmar Hilliges

In this paper, we propose VariTex - to the best of our knowledge the first method that learns a variational latent feature space of neural face textures, which allows sampling of novel identities.

Face Model

SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes

1 code implementation ICCV 2021 Xu Chen, Yufeng Zheng, Michael J. Black, Otmar Hilliges, Andreas Geiger

However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn.

The Six Hug Commandments: Design and Evaluation of a Human-Sized Hugging Robot with Visual and Haptic Perception

no code implementations19 Jan 2021 Alexis E. Block, Sammy Christen, Roger Gassert, Otmar Hilliges, Katherine J. Kuchenbecker

We followed all six tenets to create a new robotic platform, HuggieBot 2. 0, that has a soft, warm, inflated body (HuggieChest) and uses visual and haptic sensing to deliver closed-loop hugging.


Spatial Attention Improves Iterative 6D Object Pose Estimation

no code implementations5 Jan 2021 Stefan Stevsic, Otmar Hilliges

Our main insight is that after the initial pose estimate, it is important to pay attention to distinct spatial features of the object in order to improve the estimation accuracy during alignment.

6D Pose Estimation 6D Pose Estimation using RGB

Self-Learning Transformations for Improving Gaze and Head Redirection

2 code implementations NeurIPS 2020 Yufeng Zheng, Seonwook Park, Xucong Zhang, Shalini De Mello, Otmar Hilliges

Furthermore, we show that in the presence of limited amounts of real-world training data, our method allows for improvements in the downstream task of semi-supervised cross-dataset gaze estimation.

Disentanglement Gaze Estimation +1

Convolutional Autoencoders for Human Motion Infilling

1 code implementation22 Oct 2020 Manuel Kaufmann, Emre Aksan, Jie Song, Fabrizio Pece, Remo Ziegler, Otmar Hilliges

At the heart of our approach lies the idea to cast motion infilling as an inpainting problem and to train a convolutional de-noising autoencoder on image-like representations of motion sequences.

Category Level Object Pose Estimation via Neural Analysis-by-Synthesis

no code implementations ECCV 2020 Xu Chen, Zijian Dong, Jie Song, Andreas Geiger, Otmar Hilliges

Many object pose estimation algorithms rely on the analysis-by-synthesis framework which requires explicit representations of individual object instances.

Image Generation Pose Estimation

Towards End-to-end Video-based Eye-Tracking

1 code implementation ECCV 2020 Seonwook Park, Emre Aksan, Xucong Zhang, Otmar Hilliges

Estimating eye-gaze from images alone is a challenging task, in large parts due to un-observable person-specific factors.

CoSE: Compositional Stroke Embeddings

1 code implementation NeurIPS 2020 Emre Aksan, Thomas Deselaers, Andrea Tagliasacchi, Otmar Hilliges

We demonstrate qualitatively and quantitatively that our proposed approach is able to model the appearance of individual strokes, as well as the compositional structure of larger diagram drawings.

A Spatio-temporal Transformer for 3D Human Motion Prediction

1 code implementation18 Apr 2020 Emre Aksan, Manuel Kaufmann, Peng Cao, Otmar Hilliges

We propose a novel Transformer-based architecture for the task of generative modelling of 3D human motion.

Human motion prediction motion prediction

The DIDI dataset: Digital Ink Diagram data

2 code implementations20 Feb 2020 Philippe Gervais, Thomas Deselaers, Emre Aksan, Otmar Hilliges

We are releasing a dataset of diagram drawings with dynamic drawing information.

Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning

no code implementations14 Feb 2020 Sammy Christen, Lukas Jendele, Emre Aksan, Otmar Hilliges

We present HiDe, a novel hierarchical reinforcement learning architecture that successfully solves long horizon control tasks and generalizes to unseen test scenarios.

Continuous Control Decision Making +2

Content-Consistent Generation of Realistic Eyes with Style

1 code implementation8 Nov 2019 Marcel Bühler, Seonwook Park, Shalini De Mello, Xucong Zhang, Otmar Hilliges

Accurately labeled real-world training data can be scarce, and hence recent works adapt, modify or generate images to boost target datasets.

Semantic Segmentation

Structured Prediction Helps 3D Human Motion Modelling

1 code implementation ICCV 2019 Emre Aksan, Manuel Kaufmann, Otmar Hilliges

This is implemented via a hierarchy of small-sized neural networks connected analogously to the kinematic chains in the human body as well as a joint-wise decomposition in the loss function.

Human motion prediction Motion Forecasting +2

Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks

no code implementations25 Sep 2019 Lukas Jendele, Sammy Christen, Emre Aksan, Otmar Hilliges

Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction.

Continuous Control Decision Making +3

Sample Efficient Learning of Path Following and Obstacle Avoidance Behavior for Quadrotors

no code implementations28 Jun 2019 Stefan Stevsic, Tobias Naegeli, Javier Alonso-Mora, Otmar Hilliges

This enables an easy to implement learning algorithm that is robust to errors of the model used in the model predictive controller.

Imitation Learning

Demonstration-Guided Deep Reinforcement Learning of Control Policies for Dexterous Human-Robot Interaction

no code implementations27 Jun 2019 Sammy Christen, Stefan Stevsic, Otmar Hilliges

In this paper, we propose a method for training control policies for human-robot interactions such as handshakes or hand claps via Deep Reinforcement Learning.

reinforcement-learning Reinforcement Learning (RL)

Few-Shot Adaptive Gaze Estimation

1 code implementation ICCV 2019 Seonwook Park, Shalini De Mello, Pavlo Molchanov, Umar Iqbal, Otmar Hilliges, Jan Kautz

Inter-personal anatomical differences limit the accuracy of person-independent gaze estimation networks.

 Ranked #1 on Gaze Estimation on MPII Gaze (using extra training data)

Gaze Estimation Meta-Learning

Photo-Realistic Monocular Gaze Redirection Using Generative Adversarial Networks

1 code implementation ICCV 2019 Zhe He, Adrian Spurr, Xucong Zhang, Otmar Hilliges

In this work, we present a novel method to alleviate this problem by leveraging generative adversarial training to synthesize an eye image conditioned on a target gaze direction.

Gaze Estimation gaze redirection

STCN: Stochastic Temporal Convolutional Networks

1 code implementation ICLR 2019 Emre Aksan, Otmar Hilliges

Convolutional architectures have recently been shown to be competitive on many sequence modelling tasks when compared to the de-facto standard of recurrent neural networks (RNNs), while providing computational and modeling advantages due to inherent parallelism.

Unpaired Pose Guided Human Image Generation

1 code implementation8 Jan 2019 Xu Chen, Jie Song, Otmar Hilliges

This paper studies the task of full generative modelling of realistic images of humans, guided only by coarse sketch of the pose, while providing control over the specific instance or type of outfit worn by the user.

Image-to-Image Translation Translation

Monocular Neural Image Based Rendering with Continuous View Control

2 code implementations ICCV 2019 Xu Chen, Jie Song, Otmar Hilliges

The approach is self-supervised and only requires 2D images and associated view transforms for training.

Novel View Synthesis

End-to-end Learning for Graph Decomposition

no code implementations ICCV 2019 Jie Song, Bjoern Andres, Michael Black, Otmar Hilliges, Siyu Tang

The new optimization problem can be viewed as a Conditional Random Field (CRF) in which the random variables are associated with the binary edge labels of the initial graph and the hard constraints are introduced in the CRF as high-order potentials.

Multi-Person Pose Estimation

Deep Pictorial Gaze Estimation

1 code implementation ECCV 2018 Seonwook Park, Adrian Spurr, Otmar Hilliges

In this paper, we introduce a novel deep neural network architecture specifically designed for the task of gaze estimation from single eye input.

Gaze Estimation

Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings

2 code implementations12 May 2018 Seonwook Park, Xucong Zhang, Andreas Bulling, Otmar Hilliges

Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras.

Gaze Estimation

Cross-modal Deep Variational Hand Pose Estimation

1 code implementation CVPR 2018 Adrian Spurr, Jie Song, Seonwook Park, Otmar Hilliges

Furthermore, we show that our proposed method can be used without changes on depth images and performs comparably to specialized methods.

Hand Pose Estimation

DeepWriting: Making Digital Ink Editable via Deep Generative Modeling

1 code implementation25 Jan 2018 Emre Aksan, Fabrizio Pece, Otmar Hilliges

Digital ink promises to combine the flexibility and aesthetics of handwriting and the ability to process, search and edit digital text.

Handwriting generation Handwritten Word Generation +1

Guiding InfoGAN with Semi-Supervision

2 code implementations14 Jul 2017 Adrian Spurr, Emre Aksan, Otmar Hilliges

In this paper we propose a new semi-supervised GAN architecture (ss-InfoGAN) for image synthesis that leverages information from few labels (as little as 0. 22%, max.

Image Generation

Plan3D: Viewpoint and Trajectory Optimization for Aerial Multi-View Stereo Reconstruction

no code implementations25 May 2017 Benjamin Hepp, Matthias Nießner, Otmar Hilliges

We introduce a new method that efficiently computes a set of viewpoints and trajectories for high-quality 3D reconstructions in outdoor environments.

Learning Human Motion Models for Long-term Predictions

no code implementations10 Apr 2017 Partha Ghosh, Jie Song, Emre Aksan, Otmar Hilliges

Furthermore, we propose new evaluation protocols to assess the quality of synthetic motion sequences even for which no ground truth data exists.

Cannot find the paper you are looking for? You can Submit a new open access paper.