Search Results for author:
Found 107 papers, 55 papers with code
The Informed Sampler: A Discriminative Approach to Bayesian Inference in Generative Computer Vision Models
Computer vision is hard because of a large variability in lighting, shape, and texture; in addition the image signal is non-additive due to occlusion.
Consensus Message Passing for Layered Graphical Models
Generative models provide a powerful framework for probabilistic reasoning.
Permutohedral Lattice CNNs
This paper presents a convolutional layer that is able to process sparse input features.
Learning Sparse High Dimensional Filters: Image Filtering, Dense CRFs and Bilateral Neural Networks
The ability to learn more general forms of high-dimensional filters can be used in several diverse applications.
Superpixel Convolutional Networks using Bilateral Inceptions
We introduce a new 'bilateral inception' module that can be inserted in existing CNN architectures and performs bilateral filtering, at multiple feature-scales, between superpixels in an image.
Optical Flow with Semantic Segmentation and Localized Layers
Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow.
Efficient 2D and 3D Facade Segmentation using Auto-Context
This paper introduces a fast and efficient segmentation technique for 2D images and 3D point clouds of building facades.
Video Propagation Networks
We propose a 'Video Propagation Network' that processes video frames in an adaptive manner.
Ranked #72 on
Semi-Supervised Video Object Segmentation
on DAVIS 2016
Semantic Video CNNs through Representation Warping
A key insight of this work is that fast optical flow methods can be combined with many different CNN architectures for improved performance and end-to-end training.
Learning Inference Models for Computer Vision
We propose inference techniques for both generative and discriminative vision models.
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
Finally, the two input images are warped and linearly fused to form each intermediate frame.
On the Integration of Optical Flow and Action Recognition
Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better.
SPLATNet: Sparse Lattice Networks for Point Cloud Processing
We present a network architecture for processing point clouds that directly operates on a collection of points represented as a sparse set of samples in a high-dimensional lattice.
Ranked #29 on
Semantic Segmentation
on ScanNet
Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization
We present a system for training deep neural networks for object detection using synthetic images.
Switchable Temporal Propagation Network
Our approach is based on a temporal propagation network (TPN), which models the transition-related affinity between a pair of frames in a purely data-driven manner.
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation
We address the unsupervised learning of several interconnected problems in low-level vision: single view depth prediction, camera motion estimation, optical flow, and segmentation of a video into the static scene and moving regions.
Ranked #65 on
Monocular Depth Estimation
on KITTI Eigen split
Learning Superpixels With Segmentation-Aware Affinity Loss
Specifically, we propose a new loss function that takes the segmentation error into account for affinity learning.
Superpixel Sampling Networks
Superpixels provide an efficient low/mid-level representation of image data, which greatly reduces the number of image primitives for subsequent vision tasks.
Pixel-Adaptive Convolutional Neural Networks
In addition, we also demonstrate that PAC can be used as a drop-in replacement for convolution layers in pre-trained networks, resulting in consistent performance improvements.
SCOPS: Self-Supervised Co-Part Segmentation
Parts provide a good intermediate representation of objects that is robust with respect to the camera, pose and appearance variations.
Ranked #4 on
Unsupervised Keypoint Estimation
on CUB
Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
Here, we propose a new two-stream CNN architecture for semantic segmentation that explicitly wires shape information as a separate processing branch, i. e. shape stream, that processes information in parallel to the classical stream.
Ranked #24 on
Semantic Segmentation
on Cityscapes test
Learning Propagation for Arbitrarily-structured Data
We experiment with semantic segmentation networks, where we use our propagation module to jointly train on different data -- images, superpixels and point clouds.
SENSE: a Shared Encoder Network for Scene-flow Estimation
We introduce a compact network for holistic scene flow estimation, called SENSE, which shares common encoder features among four closely-related tasks: optical flow estimation, disparity estimation from stereo, occlusion estimation, and semantic segmentation.
Self-supervised Single-view 3D Reconstruction via Semantic Consistency
To the best of our knowledge, we are the first to try and solve the single-view reconstruction problem without a category-specific template mesh or semantic keypoints.
Two-shot Spatially-varying BRDF and Shape Estimation
Extensive experiments on both synthetic and real-world datasets show that our network trained on a synthetic dataset can generalize well to real-world images.
Self-Supervised Viewpoint Learning From Image Collections
Training deep neural networks to estimate the viewpoint of objects requires large labeled training datasets.
Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis
Camera captured human pose is an outcome of several sources of variation.
From Image Collections to Point Clouds with Self-supervised Shape and Pose Networks
We learn both 3D point cloud reconstruction and pose estimation networks in a self-supervised manner, making use of differentiable point cloud renderer to train with 2D supervision.
3D Object Reconstruction From A Single Image
3D Point Cloud Reconstruction
+2
Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition
Neural rendering techniques promise efficient photo-realistic image synthesis while at the same time providing rich control over scene parameters by learning the physical image formation process.
Appearance Consensus Driven Self-Supervised Human Mesh Recovery
We present a self-supervised human mesh recovery framework to infer human pose and shape from monocular images in the absence of any paired supervision.
DeepGMR: Learning Latent Gaussian Mixture Models for Registration
Point cloud registration is a fundamental problem in 3D computer vision, graphics and robotics.
Generative View Synthesis: From Single-view Semantics to Novel-view Images
We propose to push the envelope further, and introduce Generative View Synthesis (GVS), which can synthesize multiple photorealistic views of a scene given a single semantic map.
Improving Deep Stereo Network Generalization with Geometric Priors
End-to-end deep learning methods have advanced stereo vision in recent years and obtained excellent results when the training and test data are similar.
NeRD: Neural Reflectance Decomposition from Image Collections
This problem is inherently more challenging when the illumination is not a single light source under laboratory conditions but is instead an unconstrained environmental illumination.
Ranked #5 on
Image Relighting
on Stanford-ORB
Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image
We introduce the problem of perpetual view generation - long-range generation of novel views corresponding to an arbitrarily long camera trajectory given a single image.
Leveraging affinity cycle consistency to isolate factors of variation in learned representations
In this work, we operate in the setting where limited information is known about the data in the form of groupings, or set membership, and the task is to learn representations which isolate the factors of variation that are common across the groupings.
Learning ABCs: Approximate Bijective Correspondence for isolating factors of variation with weak supervision
We propose a novel algorithm that utilizes a weak form of supervision where the data is partitioned into sets according to certain inactive (common) factors of variation which are invariant across elements of each set.
Adaptive Prototype Learning and Allocation for Few-Shot Segmentation
By integrating the SGC and GPA together, we propose the Adaptive Superpixel-guided Network (ASGNet), which is a lightweight model and adapts to object scale and shape variation.
Ranked #58 on
Few-Shot Semantic Segmentation
on COCO-20i (5-shot)
Decoupled Dynamic Filter Networks
Inspired by recent advances in attention, DDF decouples a depth-wise dynamic filter into spatial and channel dynamic filters.
Ranked #13 on
Semantic Segmentation
on MCubeS
AutoFlow: Learning a Better Training Set for Optical Flow
Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications.
LASR: Learning Articulated Shape Reconstruction from a Monocular Video
Remarkable progress has been made in 3D reconstruction of rigid structures from a video or a collection of images.
Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold
Single image pose estimation is a fundamental problem in many vision and robotics tasks, and existing deep learning approaches suffer by not completely modeling and handling: i) uncertainty about the predictions, and ii) symmetric objects with multiple (sometimes infinite) correct poses.
Discovering 3D Parts from Image Collections
Reasoning 3D shapes from 2D images is an essential yet challenging task, especially when only single-view images are at our disposal.
Generalize then Adapt: Source-Free Domain Adaptive Semantic Segmentation
Unsupervised domain adaptation (DA) has gained substantial interest in semantic segmentation.
Ranked #4 on
Domain Generalization
on GTA5-to-Cityscapes
SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting
We present SLIDE, a modular and unified system for single image 3D photography that uses a simple yet effective soft layering strategy to better preserve appearance details in novel views.
Approximate Bijective Correspondence for isolating factors of variation
We propose a novel algorithm that relies on a weak form of supervision where the data is partitioned into sets according to certain \textit{inactive} factors of variation.
ViDT: An Efficient and Effective Fully Transformer-based Object Detector
Transformers are transforming the landscape of computer vision, especially for recognition tasks.
Ranked #12 on
Object Detection
on COCO 2017 val
Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition
Decomposing a scene into its shape, reflectance and illumination is a fundamental problem in computer vision and graphics.
ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction
The surface embeddings are implemented as coordinate-based MLPs that are fit to each video via consistency and contrastive reconstruction losses. Experimental results show that ViSER compares favorably against prior work on challenging videos of humans with loose clothing and unusual poses as well as animals videos from DAVIS and YTVOS.
Robust Visual Reasoning via Language Guided Neural Module Networks
Neural module networks (NMN) are a popular approach for solving multi-modal tasks such as visual question answering (VQA) and visual referring expression recognition (REF).
SOMSI: Spherical Novel View Synthesis With Soft Occlusion Multi-Sphere Images
We propose a novel MSI representation called Soft Occlusion MSI (SOMSI) that enables modelling high-dimensional appearance features in MSI while retaining the fast rendering times of a standard MSI.
Amplitude Spectrum Transformation for Open Compound Domain Adaptive Semantic Segmentation
However, we find that latent features derived from the Fourier-based amplitude spectrum of deep CNN features hold a more tractable mapping with domain discrimination.
Uncertainty-Aware Adaptation for Self-Supervised 3D Human Pose Estimation
The advances in monocular 3D human pose estimation are dominated by supervised techniques that require large-scale 2D/3D pose annotations.
Ranked #8 on
Unsupervised 3D Human Pose Estimation
on Human3.6M
Monocular 3D Human Pose Estimation
Unsupervised 3D Human Pose Estimation
+2
Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery
Articulation-centric 2D/3D pose supervision forms the core training objective in most existing 3D human pose estimation techniques.
Non-Local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation
To this end, we cast 3D pose learning as a self-supervised adaptation problem that aims to transfer the task knowledge from a labeled source domain to a completely unpaired target.
Ranked #5 on
Unsupervised 3D Human Pose Estimation
on Human3.6M
LEAD: Self-Supervised Landmark Estimation by Aligning Distributions of Feature Similarity
Existing works in self-supervised landmark detection are based on learning dense (pixel-level) feature representations from an image, which are further used to learn landmarks in a semi-supervised manner.
An Extendable, Efficient and Effective Transformer-based Object Detector
Transformers have been widely used in numerous vision problems especially for visual recognition and detection.
Planes vs. Chairs: Category-guided 3D shape learning without any 3D cues
We present a novel 3D shape reconstruction method which learns to predict an implicit 3D shape representation from a single RGB image.
SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections
Our method works on in-the-wild online image collections of an object and produces relightable 3D assets for several use-cases such as AR/VR.
Balancing Discriminability and Transferability for Source-Free Domain Adaptation
Conventional domain adaptation (DA) techniques aim to improve domain transferability by learning domain-invariant representations; while concurrently preserving the task-discriminability knowledge gathered from the labeled source data.
LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery
In this work, we propose a practical problem setting to estimate 3D pose and shape of animals given only a few (10-30) in-the-wild images of a particular animal species (say, horse).
Concurrent Subsidiary Supervision for Unsupervised Source-Free Domain Adaptation
The prime challenge in unsupervised domain adaptation (DA) is to mitigate the domain shift between the source and target domains.
Source-Free Domain Adaptation
Unsupervised Domain Adaptation
Hierarchical Semantic Regularization of Latent Spaces in StyleGANs
The quality of the generated images is predicated on two assumptions; (a) The richness of the hierarchical representations learnt by the generator, and, (b) The linearity and smoothness of the style spaces.
Improving GANs for Long-Tailed Data through Group Spectral Regularization
Deep long-tailed learning aims to train useful deep networks on practical, real-world imbalanced distributions, wherein most labels of the tail classes are associated with a few samples.
Ranked #1 on
Image Generation
on LSUN
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel photorealistic images of the subject contextualized in different scenes.
CPL: Counterfactual Prompt Learning for Vision and Language Models
Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP.
Subsidiary Prototype Alignment for Universal Domain Adaptation
Universal Domain Adaptation (UniDA) deals with the problem of knowledge transfer between two datasets with domain-shift as well as category-shift.
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis
In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions.
MetaCLUE: Towards Comprehensive Visual Metaphors Research
Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual metaphor.
Hi-LASSIE: High-Fidelity Articulated Shape and Skeleton Discovery from Sparse Image Ensemble
Automatically estimating 3D skeleton, shape, camera viewpoints, and part articulation from sparse in-the-wild image ensembles is a severely under-constrained and challenging problem.
Debiasing Vision-Language Models via Biased Prompts
Machine learning models have been shown to inherit biases from their training datasets.
Polynomial Neural Fields for Subband Decomposition and Manipulation
We use this framework to design Fourier PNFs, which match state-of-the-art performance in signal representation tasks that use neural fields.
LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Grounding
A key step to acquire this skill is to identify what part of the object affords each action, which is called affordance grounding.
DreamBooth3D: Subject-Driven Text-to-3D Generation
We present DreamBooth3D, an approach to personalize text-to-3D generative models from as few as 3-6 casually captured images of a subject.
ASIC: Aligning Sparse in-the-wild Image Collections
We present a self-supervised technique that directly optimizes on a sparse collection of images of a particular object/object category to obtain consistent dense correspondences across the collection.
NoisyTwins: Class-Consistent and Diverse Image Generation through StyleGANs
We find that one reason for degradation is the collapse of latents for each class in the $\mathcal{W}$ latent space.
Ranked #1 on
Conditional Image Generation
on ImageNet-LT
ShapeClipper: Scalable 3D Shape Learning from Single-View Images via Geometric and CLIP-based Consistency
We present ShapeClipper, a novel method that reconstructs 3D object shapes from real-world single-view RGB images.
ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimation
We propose a new dataset and a novel approach to learning hand-object interaction priors for hand and articulated object pose estimation.
Discriminative Diffusion Models as Few-shot Vision and Language Learners
Diffusion models, such as Stable Diffusion, have shown incredible performance on text-to-image generation.
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness.
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
Text-to-image diffusion models have made significant advances in generating and editing high-quality images.
Ranked #3 on
Semantic correspondence
on SPair-71k
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models
Image ad understanding is a crucial task with wide real-world applications.
Background Prompting for Improved Object Depth
To infer object depth on a real image, we place the segmented object into the learned background prompt and run off-the-shelf depth networks.
LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs
Consequently, there is growing interest in extending NeRF models to jointly optimize camera poses and scene representation, which offers an alternative to off-the-shelf SfM pipelines which have well-understood failure modes.
HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models
By composing these weights into the diffusion model, coupled with fast finetuning, HyperDreamBooth can generate a person's face in various contexts and styles, with high subject details while also preserving the model's crucial knowledge of diverse styles and semantic modifications.
OmniControl: Control Any Joint at Any Time for Human Motion Generation
We present a novel approach named OmniControl for incorporating flexible spatial control signals into a text-conditioned human motion generation model based on the diffusion process.
ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs
Experiments on a wide range of subject and style combinations show that ZipLoRA can generate compelling results with meaningful improvements over baselines in subject and style fidelity while preserving the ability to recontextualize.
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
We then explore the impact of finetuning our base model on high-quality data and train a text-to-video model that is competitive with closed-source video generation.
Exploring Attribute Variations in Style-based GANs using Diffusion Models
We capitalize on disentangled latent spaces of pretrained GANs and train a Denoising Diffusion Probabilistic Model (DDPM) to learn the latent distribution for diverse edits.
Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence
This paper identifies the importance of being geometry-aware for semantic correspondence and reveals a limitation of the features of current foundation models under simple post-processing.
Ranked #1 on
Semantic correspondence
on PF-PASCAL
One-Shot Open Affordance Learning with Foundation Models
We introduce One-shot Open Affordance Learning (OOAL), where a model is trained with just one example per base object category, but is expected to identify novel objects and affordances.
UniGS: Unified Representation for Image Generation and Segmentation
On the other hand, the progressive dichotomy module can efficiently decode the synthesized colormap to high-quality entity-level masks in a depth-first binary search without knowing the cluster numbers.
Alchemist: Parametric Control of Material Properties with Diffusion Models
We propose a method to control material attributes of objects like roughness, metallic, albedo, and transparency in real images.
NeRFiller: Completing Scenes via Generative 3D Inpainting
In contrast to related works, we focus on completing scenes rather than deleting foreground objects, and our approach does not require tight 2D object masks or text.
HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models
We also develop an affordance prediction diffusion model (APDM) to predict the contacting area between the human and object during the interactions driven by the textual prompt.
DiffusionLight: Light Probes for Free by Painting a Chrome Ball
To address this problem, we leverage diffusion models trained on billions of standard images to render a chrome ball into the input image.
ZeroShape: Regression-based Zero-shot Shape Reconstruction
In contrast, the traditional approach to this problem is regression-based, where deterministic models are trained to directly regress the object shape.
Dress-Me-Up: A Dataset & Method for Self-Supervised 3D Garment Retargeting
We propose a novel self-supervised framework for retargeting non-parameterized 3D garments onto 3D human avatars of arbitrary shapes and poses, enabling 3D virtual try-on (VTON).
SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild
We present SHINOBI, an end-to-end framework for the reconstruction of shape, material, and illumination from object images captured with varying lighting, pose, and background.
TripoSR: Fast 3D Object Reconstruction from a Single Image
This technical report introduces TripoSR, a 3D reconstruction model leveraging transformer architecture for fast feed-forward 3D generation, producing 3D mesh from a single image in under 0. 5 seconds.
3D Generation
3D Object Reconstruction From A Single Image
+2
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion
In this work, we propose SV3D that adapts image-to-video diffusion model for novel multi-view synthesis and 3D generation, thereby leveraging the generalization and multi-view consistency of the video models, while further adding explicit camera control for NVS.
WordRobe: Text-Guided Generation of Textured 3D Garments
We achieve this by first learning a latent representation of 3D garments using a novel coarse-to-fine training strategy and a loss for latent disentanglement, promoting better latent interpolation.
3D Congealing: 3D-Aware Image Alignment in the Wild
The framework optimizes for the canonical representation together with the pose for each input image, and a per-image coordinate map that warps 2D pixel coordinates to the 3D canonical frame to account for the shape matching.
MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation
We present MVD-Fusion: a method for single-view 3D inference via generative modeling of multi-view-consistent RGB-D images.
ZeST: Zero-Shot Material Transfer from a Single Image
We propose ZeST, a method for zero-shot material transfer to an object in the input image given a material exemplar image.
Probing the 3D Awareness of Visual Foundation Models
Given that such models can classify, delineate, and localize objects in 2D, we ask whether they also represent their 3D structure?
Shaping Realities: Enhancing 3D Generative AI with Fabrication Constraints
This workshop paper highlights the limitations of generative AI tools in translating digital creations into the physical world and proposes new augmentations to generative AI tools for creating physically viable 3D models.
