Search Results for author: Adam Kortylewski

Found 67 papers, 35 papers with code

ImageNet3D: Towards General-Purpose Object-Level 3D Understanding

1 code implementation13 Jun 2024 Wufei Ma, Guanning Zeng, Guofeng Zhang, Qihao Liu, Letian Zhang, Adam Kortylewski, Yaoyao Liu, Alan Yuille

A vision model with general-purpose object-level 3D understanding should be capable of inferring both 2D (e. g., class name and bounding box) and 3D information (e. g., 3D location and 3D viewpoint) for arbitrary rigid objects in natural images.

Image Captioning Linear Probing Object-Level 3D Awareness +2

FaceGPT: Self-supervised Learning to Chat about 3D Human Faces

no code implementations11 Jun 2024 Haoran Wang, Mohit Mendiratta, Christian Theobalt, Adam Kortylewski

We introduce FaceGPT, a self-supervised learning framework for Large Vision-Language Models (VLMs) to reason about 3D human faces from images and text.

3D Face Reconstruction Face Model +2

DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data

1 code implementation CVPR 2024 Qihao Liu, Yi Zhang, Song Bai, Adam Kortylewski, Alan Yuille

Unlike recent 3D generative models that rely on clean and well-aligned 3D data, limiting them to single or few-class generation, our model is directly trained on extensive noisy and unaligned `in-the-wild' 3D assets, mitigating the key challenge (i. e., data scarcity) in large-scale 3D generation.

3D Generation Text to 3D

Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering

1 code implementation2 Jun 2024 Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, Alan Yuille

To demonstrate the importance of an explicit 4D dynamics representation of the scenes in understanding world dynamics, we further propose NS-4Dynamics, a Neural-Symbolic model for reasoning on 4D Dynamics properties under explicit scene representation from videos.

counterfactual Counterfactual Reasoning +3

Evolutive Rendering Models

no code implementations27 May 2024 Fangneng Zhan, Hanxue Liang, Yifan Wang, Michael Niemeyer, Michael Oechsle, Adam Kortylewski, Cengiz Oztireli, Gordon Wetzstein, Christian Theobalt

Central to this framework is the development of differentiable versions of these rendering elements, allowing for effective gradient backpropagation from the final rendering objectives.

Learning a Category-level Object Pose Estimator without Pose Annotations

no code implementations8 Apr 2024 Fengrui Tian, Yaoyao Liu, Adam Kortylewski, Yueqi Duan, Shaoyi Du, Alan Yuille, Angtian Wang

Instead of using manually annotated images, we leverage diffusion models (e. g., Zero-1-to-3) to generate a set of images under controlled pose differences and propose to learn our object pose estimator with those images.

Object Pose Estimation

A Bayesian Approach to OOD Robustness in Image Classification

1 code implementation CVPR 2024 Prakhar Kaushik, Adam Kortylewski, Alan Yuille

This enables us to learn a transitional dictionary of vMF kernels that are intermediate between the source and target domains and train the generative model on this dictionary using the annotations on the source domain, followed by iterative refinement.

 Ranked #1 on Unsupervised Domain Adaptation on OOD-CV (Accuracy (Top-1) metric)

Image Classification Unsupervised Domain Adaptation

Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation

no code implementations19 Jan 2024 Prakhar Kaushik, Aayush Mishra, Adam Kortylewski, Alan Yuille

We focus on individual locally robust mesh vertex features and iteratively update them based on their proximity to corresponding features in the target domain even when the global pose is not correct.

Pose Estimation Unsupervised Domain Adaptation

Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos

no code implementations CVPR 2024 Leonhard Sommer, Artur Jesslen, Eddy Ilg, Adam Kortylewski

In a second step the canonical poses and reconstructed meshes enable us to train a model for 3D pose estimation from a single image.

3D Pose Estimation

Relightable Neural Actor with Intrinsic Decomposition and Pose Control

no code implementations18 Dec 2023 Diogo Luvizon, Vladislav Golyanik, Adam Kortylewski, Marc Habermann, Christian Theobalt

Creating a digital human avatar that is relightable, drivable, and photorealistic is a challenging and important problem in Vision and Graphics.

ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering

1 code implementation CVPR 2024 Haokai Pang, Heming Zhu, Adam Kortylewski, Christian Theobalt, Marc Habermann

Real-time rendering of photorealistic and controllable human avatars stands as a cornerstone in Computer Vision and Graphics.

Prompt-Based Exemplar Super-Compression and Regeneration for Class-Incremental Learning

1 code implementation30 Nov 2023 Ruxiao Duan, Yaoyao Liu, Jieneng Chen, Adam Kortylewski, Alan Yuille

Replay-based methods in class-incremental learning (CIL) have attained remarkable success, as replaying the exemplars of old classes can significantly mitigate catastrophic forgetting.

Class Incremental Learning Data Augmentation +1

DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields

no code implementations18 Nov 2023 Yu Chi, Fangneng Zhan, Sibo Wu, Christian Theobalt, Adam Kortylewski

The generated data is applicable across various computer vision tasks, including video segmentation and 3D point cloud segmentation.

Decoder Point Cloud Segmentation +3

3D-Aware Visual Question Answering about Parts, Poses and Occlusions

2 code implementations NeurIPS 2023 Xingrui Wang, Wufei Ma, Zhuowan Li, Adam Kortylewski, Alan Yuille

In this work, we introduce the task of 3D-aware VQA, which focuses on challenging questions that require a compositional reasoning over the 3D structure of visual scenes.

Question Answering Visual Question Answering

3D-Aware Neural Body Fitting for Occlusion Robust 3D Human Pose Estimation

1 code implementation ICCV 2023 Yi Zhang, Pengliang Ji, Angtian Wang, Jieru Mei, Adam Kortylewski, Alan Yuille

Motivated by the recent success of generative models in rigid object pose estimation, we propose 3D-aware Neural Body Fitting (3DNBF) - an approximate analysis-by-synthesis approach to 3D human pose estimation with SOTA performance and occlusion robustness.

3D Human Pose Estimation Contrastive Learning

Generating Images with 3D Annotations Using Diffusion Models

no code implementations13 Jun 2023 Wufei Ma, Qihao Liu, Jiahao Wang, Angtian Wang, Xiaoding Yuan, Yi Zhang, Zihao Xiao, Guofeng Zhang, Beijia Lu, Ruxiao Duan, Yongrui Qi, Adam Kortylewski, Yaoyao Liu, Alan Yuille

With explicit 3D geometry control, we can easily change the 3D structures of the objects in the generated images and obtain ground-truth 3D annotations automatically.

3D Pose Estimation Style Transfer

AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars

no code implementations1 Jun 2023 Mohit Mendiratta, Xingang Pan, Mohamed Elgharib, Kartik Teotia, Mallikarjun B R, Ayush Tewari, Vladislav Golyanik, Adam Kortylewski, Christian Theobalt

Our method edits the full head in a canonical space, and then propagates these edits to remaining time steps via a pretrained deformation network.

Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Search

no code implementations1 Jun 2023 Qihao Liu, Adam Kortylewski, Yutong Bai, Song Bai, Alan Yuille

(2) We find regions in the latent space that lead to distorted images independent of the text prompt, suggesting that parts of the latent space are not well-structured.

Adversarial Attack Efficient Exploration +1

Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis

no code implementations31 May 2023 Angtian Wang, Wufei Ma, Alan Yuille, Adam Kortylewski

Human vision demonstrates higher robustness than current AI algorithms under out-of-distribution scenarios.

Robust Category-Level 3D Pose Estimation from Synthetic Data

no code implementations25 May 2023 Jiahao Yang, Wufei Ma, Angtian Wang, Xiaoding Yuan, Alan Yuille, Adam Kortylewski

In this work, we aim to narrow the performance gap between models trained on synthetic data and few real images and fully supervised models trained on large-scale data.

3D Pose Estimation 3D Reconstruction +4

Robust 3D-aware Object Classification via Discriminative Render-and-Compare

no code implementations24 May 2023 Artur Jesslen, Guofeng Zhang, Angtian Wang, Alan Yuille, Adam Kortylewski

Using differentiable rendering, we estimate the 3D object pose by minimizing the reconstruction error between the mesh and the feature representation of the target image.

Classification Image Classification +2

General Neural Gauge Fields

1 code implementation5 May 2023 Fangneng Zhan, Lingjie Liu, Adam Kortylewski, Christian Theobalt

In this work, we extend this problem to a general paradigm with a taxonomy of discrete \& continuous cases, and develop a learning framework to jointly optimize gauge transformations and neural fields.

Representation Learning

PoseExaminer: Automated Testing of Out-of-Distribution Robustness in Human Pose and Shape Estimation

1 code implementation CVPR 2023 Qihao Liu, Adam Kortylewski, Alan Yuille

We introduce a learning-based testing method, termed PoseExaminer, that automatically diagnoses HPS algorithms by searching over the parameter space of human pose images to find the failure modes.

Multi-agent Reinforcement Learning

Scene-Aware 3D Multi-Human Motion Capture from a Single Camera

1 code implementation12 Jan 2023 Diogo Luvizon, Marc Habermann, Vladislav Golyanik, Adam Kortylewski, Christian Theobalt

In this work, we consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera.

Position

State of the Art in Dense Monocular Non-Rigid 3D Reconstruction

no code implementations27 Oct 2022 Edith Tretschk, Navami Kairanda, Mallikarjun B R, Rishabh Dabral, Adam Kortylewski, Bernhard Egger, Marc Habermann, Pascal Fua, Christian Theobalt, Vladislav Golyanik

3D reconstruction of deformable (or non-rigid) scenes from a set of monocular 2D image observations is a long-standing and actively researched area of computer vision and graphics.

3D Reconstruction

HandFlow: Quantifying View-Dependent 3D Ambiguity in Two-Hand Reconstruction with Normalizing Flow

no code implementations4 Oct 2022 Jiayi Wang, Diogo Luvizon, Franziska Mueller, Florian Bernard, Adam Kortylewski, Dan Casas, Christian Theobalt

Through this, we demonstrate the quality of our probabilistic reconstruction and show that explicit ambiguity modeling is better-suited for this challenging problem.

valid

VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis

1 code implementation30 May 2022 Angtian Wang, Peng Wang, Jian Sun, Adam Kortylewski, Alan Yuille

The Gaussian reconstruction kernels have been proposed by Westover (1990) and studied by the computer graphics community back in the 90s, which gives an alternative representation of object 3D geometry from meshes and point clouds.

Pose Estimation

Multimodal Image Synthesis and Editing: The Generative AI Era

2 code implementations27 Dec 2021 Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, Lingjie Liu, Adam Kortylewski, Christian Theobalt, Eric Xing

With superb power in modeling the interaction among multimodal information, multimodal image synthesis and editing has become a hot research topic in recent years.

Image Generation

PartImageNet: A Large, High-Quality Dataset of Parts

1 code implementation2 Dec 2021 Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, Alan Yuille

To help address this problem, we propose PartImageNet, a large, high-quality dataset with part segmentation annotations.

Activity Recognition Few-Shot Learning +6

OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images

no code implementations29 Nov 2021 Bingchen Zhao, Shaozuo Yu, Wufei Ma, Mingxin Yu, Shenxiao Mei, Angtian Wang, Ju He, Alan Yuille, Adam Kortylewski

One reason is that existing robustness benchmarks are limited, as they either rely on synthetic data or ignore the effects of individual nuisance factors.

3D Pose Estimation Benchmarking +5

Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose

1 code implementation NeurIPS 2021 Angtian Wang, Shenxiao Mei, Alan Yuille, Adam Kortylewski

The model is initialized from a few labelled images and is subsequently used to synthesize feature representations of unseen 3D views.

3D Pose Estimation Few-Shot Learning

Simulated Adversarial Testing of Face Recognition Models

no code implementations CVPR 2022 Nataniel Ruiz, Adam Kortylewski, Weichao Qiu, Cihang Xie, Sarah Adel Bargal, Alan Yuille, Stan Sclaroff

In this work, we propose a framework for learning how to test machine learning algorithms using simulators in an adversarial manner in order to find weaknesses in the model before deploying it in critical scenarios.

BIG-bench Machine Learning Face Recognition

Rethinking Re-Sampling in Imbalanced Semi-Supervised Learning

1 code implementation1 Jun 2021 Ju He, Adam Kortylewski, Shaokang Yang, Shuai Liu, Cheng Yang, Changhu Wang, Alan Yuille

In particular, we decouple the training of the representation and the classifier, and systematically investigate the effects of different data re-sampling techniques when training the whole network including a classifier as well as fine-tuning the feature extractor only.

A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation

1 code implementation ICCV 2021 Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, Xiaolong Wang

To deal with the large shape variance, we introduce Articulated Signed Distance Functions (A-SDF) to represent articulated shapes with a disentangled latent space, where we have separate codes for encoding shape and articulation.

Test-time Adaptation

TransFG: A Transformer Architecture for Fine-grained Recognition

2 code implementations14 Mar 2021 Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang

Fine-grained visual classification (FGVC) which aims at recognizing objects from subcategories is a very challenging task due to the inherently subtle inter-class differences.

Fine-Grained Image Classification

Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping

1 code implementation22 Feb 2021 Prakhar Kaushik, Alex Gain, Adam Kortylewski, Alan Yuille

Additionally, current approaches that deal with forgetting ignore the problem of catastrophic remembering, i. e. the worsening ability to discriminate between data from different tasks.

Continual Learning

NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation

1 code implementation ICLR 2021 Angtian Wang, Adam Kortylewski, Alan Yuille

Using differentiable rendering we estimate the 3D object pose by minimizing the reconstruction error between NeMo and the feature representation of the target image.

3D Pose Estimation Contrastive Learning

CORL: Compositional Representation Learning for Few-Shot Classification

no code implementations28 Jan 2021 Ju He, Adam Kortylewski, Alan Yuille

In particular, during meta-learning, we train a knowledge base that consists of a dictionary of component representations and a dictionary of component activation maps that encode common spatial activation patterns of components.

Classification Few-Shot Image Classification +3

Unsupervised Part Discovery via Feature Alignment

no code implementations1 Dec 2020 Mengqi Guo, Yutong Bai, Zhishuai Zhang, Adam Kortylewski, Alan Yuille

Specifically, given a training image, we find a set of similar images that show instances of the same object category in the same pose, through an affine alignment of their corresponding feature maps.

Object Object Recognition

Robustness Out of the Box: Compositional Representations Naturally Defend Against Black-Box Patch Attacks

no code implementations1 Dec 2020 Christian Cosgrove, Adam Kortylewski, Chenglin Yang, Alan Yuille

Second, we find that compositional deep networks, which have part-based representations that lead to innate robustness to natural occlusion, are robust to patch attacks on PASCAL3D+ and the German Traffic Sign Recognition Benchmark, without adversarial training.

Traffic Sign Recognition

Amodal Segmentation through Out-of-Task and Out-of-Distribution Generalization with a Bayesian Model

1 code implementation CVPR 2022 Yihong Sun, Adam Kortylewski, Alan Yuille

Moreover, by leveraging an outlier process, Bayesian models can further generalize out-of-distribution to segment partially occluded objects and to predict their amodal object boundaries.

Amodal Instance Segmentation Object +2

Compositional Convolutional Neural Networks: A Robust and Interpretable Model for Object Recognition under Occlusion

no code implementations28 Jun 2020 Adam Kortylewski, Qing Liu, Angtian Wang, Yihong Sun, Alan Yuille

The structure of the compositional model enables CompositionalNets to decompose images into objects and context, as well as to further decompose object representations in terms of individual parts and the objects' pose.

Image Classification object-detection +2

Robust Object Detection under Occlusion with Context-Aware CompositionalNets

no code implementations CVPR 2020 Angtian Wang, Yihong Sun, Adam Kortylewski, Alan Yuille

In this work, we propose to overcome two limitations of CompositionalNets which will enable them to detect partially occluded objects: 1) CompositionalNets, as well as other DCNN architectures, do not explicitly separate the representation of the context from the object itself.

Object object-detection +1

Compositional Convolutional Neural Networks: A Deep Architecture with Innate Robustness to Partial Occlusion

1 code implementation CVPR 2020 Adam Kortylewski, Ju He, Qing Liu, Alan Yuille

Inspired by the success of compositional models at classifying partially occluded objects, we propose to integrate compositional models and DCNNs into a unified deep model with innate robustness to partial occlusion.

General Classification

Localizing Occluders with Compositional Convolutional Networks

no code implementations18 Nov 2019 Adam Kortylewski, Qing Liu, Huiyu Wang, Zhishuai Zhang, Alan Yuille

Our experimental results demonstrate that the proposed extensions increase the model's performance at localizing occluders as well as at classifying partially occluded objects.

TDAPNet: Prototype Network with Recurrent Top-Down Attention for Robust Object Classification under Partial Occlusion

no code implementations9 Sep 2019 Mingqing Xiao, Adam Kortylewski, Ruihai Wu, Siyuan Qiao, Wei Shen, Alan Yuille

Despite deep convolutional neural networks' great success in object classification, it suffers from severe generalization performance drop under occlusion due to the inconsistency between training and testing data.

General Classification Object +1

Combining Compositional Models and Deep Networks For Robust Object Classification under Occlusion

no code implementations28 May 2019 Adam Kortylewski, Qing Liu, Huiyu Wang, Zhishuai Zhang, Alan Yuille

In this work, we combine DCNNs and compositional object models to retain the best of both approaches: a discriminative model that is robust to partial occlusion and mask attacks.

General Classification Image Classification +1

SkelNetOn 2019: Dataset and Challenge on Deep Learning for Geometric Shape Understanding

no code implementations21 Mar 2019 Ilke Demir, Camilla Hahn, Kathryn Leonard, Geraldine Morin, Dana Rahbani, Athina Panotopoulou, Amelie Fondevilla, Elena Balashova, Bastien Durix, Adam Kortylewski

We present SkelNetOn 2019 Challenge and Deep Learning for Geometric Shape Understanding workshop to utilize existing and develop novel deep learning architectures for shape understanding.

Can Synthetic Faces Undo the Damage of Dataset Bias to Face Recognition and Facial Landmark Detection?

1 code implementation19 Nov 2018 Adam Kortylewski, Bernhard Egger, Andreas Morel-Forster, Andreas Schneider, Thomas Gerig, Clemens Blumer, Corius Reyneke, Thomas Vetter

We observe the following positive effects for face recognition and facial landmark detection tasks: 1) Priming with synthetic face images improves the performance consistently across all benchmarks because it reduces the negative effects of biases in the training data.

Data Augmentation Face Model +3

Informed MCMC with Bayesian Neural Networks for Facial Image Analysis

no code implementations19 Nov 2018 Adam Kortylewski, Mario Wieser, Andreas Morel-Forster, Aleksander Wieczorek, Sonali Parbhoo, Volker Roth, Thomas Vetter

Computer vision tasks are difficult because of the large variability in the data that is induced by changes in light, background, partial occlusion as well as the varying pose, texture, and shape of objects.

Bayesian Inference valid

Training Deep Face Recognition Systems with Synthetic Data

2 code implementations16 Feb 2018 Adam Kortylewski, Andreas Schneider, Thomas Gerig, Bernhard Egger, Andreas Morel-Forster, Thomas Vetter

In our experiments with an off-the-shelf face recognition software we observe the following phenomena: 1) The amount of real training data needed to train competitive deep face recognition systems can be reduced significantly.

Face Model Face Recognition

Empirically Analyzing the Effect of Dataset Biases on Deep Face Recognition Systems

2 code implementations5 Dec 2017 Adam Kortylewski, Bernhard Egger, Andreas Schneider, Thomas Gerig, Andreas Morel-Forster, Thomas Vetter

4) We uncover a main limitation of current DCNN architectures, which is the difficulty to generalize when different identities to not share the same pose variation.

Face Recognition

Greedy Structure Learning of Hierarchical Compositional Models

no code implementations CVPR 2019 Adam Kortylewski, Aleksander Wieczorek, Mario Wieser, Clemens Blumer, Sonali Parbhoo, Andreas Morel-Forster, Volker Roth, Thomas Vetter

In this work, we consider the problem of learning a hierarchical generative model of an object from a set of images which show examples of the object in the presence of variable background clutter.

Object Transfer Learning

Cannot find the paper you are looking for? You can Submit a new open access paper.