Search Results for author: Niloy J. Mitra

Found 60 papers, 29 papers with code

Points2Surf Learning Implicit Surfaces from Point Clouds

1 code implementation ECCV 2020 Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Niloy J. Mitra, Michael Wimmer

Hence, deep learning based methods have recently been proposed to produce complete surfaces, even from partial scans.

Surface Reconstruction

GOEnFusion: Gradient Origin Encodings for 3D Forward Diffusion Models

no code implementations14 Dec 2023 Animesh Karnewar, Andrea Vedaldi, Niloy J. Mitra, David Novotny

The recently introduced Forward-Diffusion method allows to train a 3D diffusion model using only 2D images for supervision.

LooseControl: Lifting ControlNet for Generalized Depth Conditioning

no code implementations5 Dec 2023 Shariq Farooq Bhat, Niloy J. Mitra, Peter Wonka

We present LooseControl to allow generalized depth conditioning for diffusion-based image generation.

Attribute Image Generation

Evaluating VLMs for Score-Based, Multi-Probe Annotation of 3D Objects

no code implementations29 Nov 2023 Rishabh Kabra, Loic Matthey, Alexander Lerchner, Niloy J. Mitra

Unlabeled 3D objects present an opportunity to leverage pretrained vision language models (VLMs) on a range of annotation tasks -- from describing object semantics to physical properties.

In-Context Learning Language Modelling +2

CADTalk: An Algorithm and Benchmark for Semantic Commenting of CAD Programs

no code implementations28 Nov 2023 Haocheng Yuan, Jing Xu, Hao Pan, Adrien Bousseau, Niloy J. Mitra, Changjian Li

CAD programs are a popular way to compactly encode shapes as a sequence of operations that are easy to parametrically modify.

Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features

1 code implementation28 Nov 2023 Niladri Shekhar Dutt, Sanjeev Muralikrishnan, Niloy J. Mitra

We present Diff3F as a simple, robust, and class-agnostic feature descriptor that can be computed for untextured input shapes (meshes or point clouds).

3D Dense Shape Correspondence 3D Part Segmentation

Efficient Gradient Estimation via Adaptive Sampling and Importance Sampling

no code implementations24 Nov 2023 Corentin Salaün, Xingchang Huang, Iliyan Georgiev, Niloy J. Mitra, Gurprit Singh

Nevertheless, existing algorithms have struggled to efficiently integrate importance sampling into machine learning frameworks.

ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context

no code implementations15 Oct 2023 Binglun Wang, Niladri Shekhar Dutt, Niloy J. Mitra

We evaluate our setup on a variety of examples to demonstrate appearance and geometric edits and report 10-30x speedup over concurrent work focusing on text-guided NeRF editing.

Neural Semantic Surface Maps

no code implementations9 Sep 2023 Luca Morreale, Noam Aigerman, Vladimir G. Kim, Niloy J. Mitra

We present an automated technique for computing a map between two genus-zero shapes, which matches semantically corresponding regions to one another.

BLiSS: Bootstrapped Linear Shape Space

no code implementations4 Sep 2023 Sanjeev Muralikrishnan, Chun-Hao Paul Huang, Duygu Ceylan, Niloy J. Mitra

Morphable models are fundamental to numerous human-centered processes as they offer a simple yet expressive shape space.

HoloFusion: Towards Photo-realistic 3D Generative Modeling

no code implementations ICCV 2023 Animesh Karnewar, Niloy J. Mitra, Andrea Vedaldi, David Novotny

Diffusion-based image generators can now produce high-quality and diverse samples, but their success has yet to fully translate to 3D generation: existing diffusion methods can either generate low-resolution but 3D consistent outputs, or detailed 2D views of 3D objects but with potential structural defects and lacking view consistency or realism.

Super-Resolution

ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives

1 code implementation9 May 2023 R. Kenny Jones, Paul Guerrero, Niloy J. Mitra, Daniel Ritchie

The discovered abstractions capture common patterns (both structural and parametric) across the dataset, so that programs rewritten with these abstractions are more compact, and expose fewer degrees of freedom.

Factored Neural Representation for Scene Understanding

no code implementations21 Apr 2023 Yu-Shiang Wong, Niloy J. Mitra

A long-standing goal in scene understanding is to obtain interpretable and editable representations that can be directly constructed from a raw monocular RGB-D video, without requiring specialized hardware setup or priors.

Novel View Synthesis Object +1

Pix2Video: Video Editing using Image Diffusion

1 code implementation ICCV 2023 Duygu Ceylan, Chun-Hao Paul Huang, Niloy J. Mitra

Our method works in two simple steps: first, we use a pre-trained structure-guided (e. g., depth) image diffusion model to perform text-guided edits on an anchor frame; then, in the key step, we progressively propagate the changes to the future frames via self-attention feature injection to adapt the core denoising step of the diffusion model.

Denoising Text Generation +1

Search for Concepts: Discovering Visual Concepts Using Direct Optimization

no code implementations25 Oct 2022 Pradyumna Reddy, Paul Guerrero, Niloy J. Mitra

Finding an unsupervised decomposition of an image into individual objects is a key step to leverage compositionality and to perform symbolic reasoning.

Motion Guided Deep Dynamic 3D Garments

1 code implementation23 Sep 2022 Meng Zhang, Duygu Ceylan, Niloy J. Mitra

Technically, we model garment dynamics, driven using the input character motion, by predicting per-frame local displacements in a canonical state of the garment that is enriched with frame-dependent skinning weights to bring the garment to the global space.

NeuForm: Adaptive Overfitting for Neural Shape Editing

no code implementations18 Jul 2022 Connor Z. Lin, Niloy J. Mitra, Gordon Wetzstein, Leonidas Guibas, Paul Guerrero

Neural representations are popular for representing shapes, as they can be learned form sensor data and used for data cleanup, model completion, shape editing, and shape synthesis.

Video2StyleGAN: Disentangling Local and Global Variations in a Video

no code implementations27 May 2022 Rameen Abdal, Peihao Zhu, Niloy J. Mitra, Peter Wonka

Image editing using a pretrained StyleGAN generator has emerged as a powerful paradigm for facial editing, providing disentangled controls over age, expression, illumination, etc.

Facial Editing

ReLU Fields: The Little Non-linearity That Could

no code implementations22 May 2022 Animesh Karnewar, Tobias Ritschel, Oliver Wang, Niloy J. Mitra

Although the MLPs are able to represent complex scenes with unprecedented quality and memory footprint, this expressive power of the MLPs, however, comes at the cost of long training and inference times.

Neural Convolutional Surfaces

no code implementations CVPR 2022 Luca Morreale, Noam Aigerman, Paul Guerrero, Vladimir G. Kim, Niloy J. Mitra

Our pipeline and architecture are designed so that disentanglement of global geometry from local details is accomplished through optimization, in a completely unsupervised manner.

Disentanglement

InsetGAN for Full-Body Image Generation

2 code implementations CVPR 2022 Anna Frühstück, Krishna Kumar Singh, Eli Shechtman, Niloy J. Mitra, Peter Wonka, Jingwan Lu

Instead of modeling this complex domain with a single GAN, we propose a novel method to combine multiple pretrained GANs, where one GAN generates a global canvas (e. g., human body) and a set of specialized GANs, or insets, focus on different parts (e. g., faces, shoes) that can be seamlessly inserted onto the global canvas.

Image Generation

ShapeFormer: Transformer-based Shape Completion via Sparse Representation

1 code implementation CVPR 2022 Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischinski, Daniel Cohen-Or, Hui Huang

We present ShapeFormer, a transformer-based network that produces a distribution of object completions, conditioned on incomplete, and possibly noisy, point clouds.

CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions

no code implementations9 Dec 2021 Rameen Abdal, Peihao Zhu, John Femiani, Niloy J. Mitra, Peter Wonka

The success of StyleGAN has enabled unprecedented semantic editing capabilities, on both synthesized and real images.

Zero-Shot Learning

Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis

no code implementations10 Nov 2021 Tuanfeng Y. Wang, Duygu Ceylan, Krishna Kumar Singh, Niloy J. Mitra

Synthesizing dynamic appearances of humans in motion plays a central role in applications such as AR/VR and video editing.

motion retargeting Video Editing

CPFN: Cascaded Primitive Fitting Networks for High-Resolution Point Clouds

1 code implementation ICCV 2021 Eric-Tuan Lê, Minhyuk Sung, Duygu Ceylan, Radomir Mech, Tamy Boubekeur, Niloy J. Mitra

We present Cascaded Primitive Fitting Networks (CPFN) that relies on an adaptive patch sampling network to assemble detection results of global and local primitive detection networks.

Vocal Bursts Intensity Prediction

MoCo-Flow: Neural Motion Consensus Flow for Dynamic Humans in Stationary Monocular Cameras

no code implementations8 Jun 2021 Xuelin Chen, Weiyu Li, Daniel Cohen-Or, Niloy J. Mitra, Baoquan Chen

In this paper, we introduce Neural Motion Consensus Flow (MoCo-Flow), a representation that models dynamic humans in stationary monocular cameras using a 4D continuous time-variant function.

Consistent Two-Flow Network for Tele-Registration of Point Clouds

1 code implementation1 Jun 2021 Zihao Yan, Zimu Yi, Ruizhen Hu, Niloy J. Mitra, Daniel Cohen-Or, Hui Huang

In this paper, we present a learning-based technique that alleviates this problem, and allows registration between point clouds, presented in arbitrary poses, and having little or even no overlap, a setting that has been referred to as tele-registration.

Vocal Bursts Valence Prediction

ShapeMOD: Macro Operation Discovery for 3D Shape Programs

1 code implementation13 Apr 2021 R. Kenny Jones, David Charatan, Paul Guerrero, Niloy J. Mitra, Daniel Ritchie

In this paper, we present ShapeMOD, an algorithm for automatically discovering macros that are useful across large datasets of 3D shape programs.

Neural Surface Maps

1 code implementation CVPR 2021 Luca Morreale, Noam Aigerman, Vladimir Kim, Niloy J. Mitra

Maps are arguably one of the most fundamental concepts used to define and operate on manifold surfaces in differentiable geometry.

Dynamic Neural Garments

no code implementations23 Feb 2021 Meng Zhang, Duygu Ceylan, Tuanfeng Wang, Niloy J. Mitra

A vital task of the wider digital human effort is the creation of realistic garments on digital avatars, both in the form of characteristic fold patterns and wrinkles in static frames as well as richness of garment dynamics under avatars' motion.

Neural Rendering

Generative Modelling of BRDF Textures from Flash Images

no code implementations23 Feb 2021 Philipp Henzler, Valentin Deschaintre, Niloy J. Mitra, Tobias Ritschel

We learn a latent space for easy capture, consistent interpolation, and efficient reproduction of visual material appearance.

Im2Vec: Synthesizing Vector Graphics without Vector Supervision

1 code implementation CVPR 2021 Pradyumna Reddy, Michael Gharbi, Michal Lukac, Niloy J. Mitra

The current alternative is to use specialized models that require explicit supervision on the vector graphics representation at training time.

Vector Graphics

Seeing Behind Objects for 3D Multi-Object Tracking in RGB-D Sequences

no code implementations CVPR 2021 Norman Müller, Yu-Shiang Wong, Niloy J. Mitra, Angela Dai, Matthias Nießner

From a sequence of RGB-D frames, we detect objects in each frame and learn to predict their complete object geometry as well as a dense correspondence mapping into a canonical space.

3D Multi-Object Tracking Object

Points2Surf: Learning Implicit Surfaces from Point Cloud Patches

2 code implementations20 Jul 2020 Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Michael Wimmer, Niloy J. Mitra

Hence, deep learning based methods have recently been proposed to produce complete surfaces, even from partial scans.

Towards a Neural Graphics Pipeline for Controllable Image Generation

no code implementations18 Jun 2020 Xuelin Chen, Daniel Cohen-Or, Baoquan Chen, Niloy J. Mitra

NGP decomposes the image into a set of interpretable appearance feature maps, uncovering direct control handles for controllable image generation.

Image Generation Neural Rendering

Learning a Neural 3D Texture Space from 2D Exemplars

1 code implementation CVPR 2020 Philipp Henzler, Niloy J. Mitra, Tobias Ritschel

We propose a generative model of 2D and 3D natural textures with diversity, visual fidelity and at high computational efficiency.

Computational Efficiency

Going Deeper with Lean Point Networks

1 code implementation CVPR 2020 Eric-Tuan Le, Iasonas Kokkinos, Niloy J. Mitra

By combining these blocks, we design wider and deeper point-based architectures.

Unsupervised Intuitive Physics from Past Experiences

no code implementations26 May 2019 Sébastien Ehrhardt, Aron Monszpart, Niloy J. Mitra, Andrea Vedaldi

We are interested in learning models of intuitive physics similar to the ones that animals use for navigation, manipulation and planning.

Meta-Learning

Unpaired Point Cloud Completion on Real Scans using Adversarial Training

2 code implementations ICLR 2020 Xuelin Chen, Baoquan Chen, Niloy J. Mitra

As 3D scanning solutions become increasingly popular, several deep learning setups have been developed geared towards that task of scan completion, i. e., plausibly filling in regions there were missed in the raw scans.

Point Cloud Completion

PointCleanNet: Learning to Denoise and Remove Outliers from Dense Point Clouds

1 code implementation4 Jan 2019 Marie-Julie Rakotosaona, Vittorio La Barbera, Paul Guerrero, Niloy J. Mitra, Maks Ovsjanikov

Point clouds obtained with 3D scanners or by image-based reconstruction techniques are often corrupted with significant amount of noise and outliers.

Denoising Surface Reconstruction

Learning a Shared Shape Space for Multimodal Garment Design

no code implementations29 Jun 2018 Tuanfeng Y. Wang, Duygu Ceylan, Jovan Popovic, Niloy J. Mitra

Designing real and virtual garments is becoming extremely demanding with rapidly changing fashion trends and increasing need for synthesizing realistic dressed digital humans for various applications.

Graphics

iMapper: Interaction-guided Joint Scene and Human Motion Mapping from Monocular Videos

no code implementations20 Jun 2018 Aron Monszpart, Paul Guerrero, Duygu Ceylan, Ersin Yumer, Niloy J. Mitra

A long-standing challenge in scene analysis is the recovery of scene arrangements under moderate to heavy occlusion, directly from monocular video.

Human-Object Interaction Detection Object

FrankenGAN: Guided Detail Synthesis for Building Mass-Models Using Style-Synchonized GANs

2 code implementations19 Jun 2018 Tom Kelly, Paul Guerrero, Anthony Steed, Peter Wonka, Niloy J. Mitra

The various GANs are synchronized to produce consistent style distributions over buildings and neighborhoods.

Graphics

Learning on the Edge: Explicit Boundary Handling in CNNs

1 code implementation8 May 2018 Carlo Innamorati, Tobias Ritschel, Tim Weyrich, Niloy J. Mitra

Convolutional neural networks (CNNs) handle the case where filters extend beyond the image boundary using several heuristics, such as zero, repeat or mean padding.

Colorization Disparity Estimation +1

Joint Material and Illumination Estimation from Photo Sets in the Wild

1 code implementation23 Oct 2017 Tuanfeng Y. Wang, Tobias Ritschel, Niloy J. Mitra

To the other hand, methods that are automatic and work on 'in the wild' Internet images, often extract only low-frequency lighting or diffuse materials.

Graphics

PCPNET: Learning Local Shape Properties from Raw Point Clouds

33 code implementations13 Oct 2017 Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, Niloy J. Mitra

In this paper, we propose PCPNet, a deep-learning based approach for estimating local 3D shape properties in point clouds.

Computational Geometry

DepthCut: Improved Depth Edge Estimation Using Multiple Unreliable Channels

no code implementations22 May 2017 Paul Guerrero, Holger Winnemöller, Wilmot Li, Niloy J. Mitra

In the context of scene understanding, a variety of methods exists to estimate different information channels from mono or stereo images, including disparity, depth, and normals.

Scene Understanding Segmentation

Learning A Physical Long-term Predictor

no code implementations1 Mar 2017 Sebastien Ehrhardt, Aron Monszpart, Niloy J. Mitra, Andrea Vedaldi

Evolution has resulted in highly developed abilities in many natural intelligences to quickly and accurately predict mechanical phenomena.

SMASH: Physics-guided Reconstruction of Collisions from Videos

1 code implementation29 Mar 2016 Aron Monszpart, Nils Thuerey, Niloy J. Mitra

Authoring even two body collisions in the real world can be difficult, as one has to get timing and the object trajectories to be correctly synchronized.

3D Reconstruction valid

RAID: A Relation-Augmented Image Descriptor

no code implementations5 Oct 2015 Paul Guerrero, Niloy J. Mitra, Peter Wonka

As humans, we regularly interpret images based on the relations between image regions.

Relation

Super 4PCS Fast Global Pointcloud Registration via Smart Indexing

1 code implementation Computer Graphics Forum 2014 Nicolas Mellado, Dror Aiger, Niloy J. Mitra

We present Super 4PCS for global pointcloud registration that is optimal, i. e., runs in linear time (in the number of data points) and is also output sensitive in the complexity of the alignment problem based on the (unknown) overlap across scan pairs.

Point Cloud Registration

SmartAnnotator: An Interactive Tool for Annotating RGBD Indoor Images

no code implementations23 Mar 2014 Yu-Shiang Wong, Hung-Kuo Chu, Niloy J. Mitra

Further, as more scenes are annotated, the system makes better suggestions based on structural and geometric priors learns from the previous annotation sessions.

Image Manipulation

Cannot find the paper you are looking for? You can Submit a new open access paper.