Search Results for author:
Found 95 papers, 34 papers with code
Muse: Text-To-Image Generation via Masked Generative Transformers
Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding.
Ranked #18 on
Text-to-Image Generation
on COCO
MetaCLUE: Towards Comprehensive Visual Metaphors Research
Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual metaphor.
Can Shadows Reveal Biometric Information?
We transfer what we learn from the synthetic data to the real data using domain adaptation in a completely unsupervised way.
Seeing 3D Objects in a Single Image via Self-Supervised Static-Dynamic Disentanglement
We separately parameterize movable and immovable scene parts via 2D neural ground plans.
Disentangling Architecture and Training for Optical Flow
Our newly trained RAFT achieves an Fl-all score of 4. 31% on KITTI 2015, more accurate than all published optical flow methods at the time of writing.
Unsupervised Semantic Segmentation by Distilling Feature Correspondences
Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation.
Ranked #1 on
Unsupervised Semantic Segmentation
on COCO-Stuff
MaskGIT: Masked Generative Image Transformer
At inference time, the model begins with generating all tokens of an image simultaneously, and then refines the image iteratively conditioned on the previous generation.
Ranked #2 on
Text-to-Image Generation
on LHQC
HSPACE: Synthetic Parametric Humans Animated in Complex Environments
We combine a hundred diverse individuals of varying ages, gender, proportions, and ethnicity, with hundreds of motions and scenes, as well as parametric variations in body shape (for a total of 1, 600 different humans), in order to generate an initial dataset of over 1 million frames.
Ranked #1 on
3D Human Pose Estimation
on HSPACE
SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting
We present SLIDE, a modular and unified system for single image 3D photography that uses a simple yet effective soft layering strategy to better preserve appearance details in novel views.
What You Can Learn by Staring at a Blank Wall
We present a passive non-line-of-sight method that infers the number of people or activity of a person from the observation of a blank wall in an unknown room.
Consistent Depth of Moving Objects in Video
We present a method to estimate depth of a dynamic scene, containing arbitrary moving objects, from an ordinary video captured with a moving camera.
THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers
We present THUNDR, a transformer-based deep neural network methodology to reconstruct the 3d pose and shape of people, given monocular RGB images.
Ranked #182 on
3D Human Pose Estimation
on Human3.6M
Toward Automatic Interpretation of 3D Plots
This paper explores the challenge of teaching a machine how to reverse-engineer the grid-marked surfaces used to represent data in 3D surface plots of two-variable functions.
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering
In this work, we propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field parameterized via a neural implicit representation.
NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties.
Omnimatte: Associating Objects and Their Effects in Video
We show results on real-world videos containing interactions between different types of subjects (cars, animals, people) and complex effects, ranging from semi-transparent elements such as smoke and reflections, to fully opaque effects such as objects attached to the subject.
LASR: Learning Articulated Shape Reconstruction from a Monocular Video
Remarkable progress has been made in 3D reconstruction of rigid structures from a video or a collection of images.
AutoFlow: Learning a Better Training Set for Optical Flow
Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications.
Explaining in Style: Training a GAN to explain a classifier in StyleSpace
A natural source for such attributes is the StyleSpace of StyleGAN, which is known to generate semantically meaningful dimensions in the image.
Axiomatic Explanations for Visual Search, Retrieval, and Similarity Learning
Visual search, recommendation, and contrastive similarity learning power technologies that impact billions of users worldwide.
Multi-Plane Program Induction with 3D Box Priors
We consider two important aspects in understanding and editing images: modeling regular, program-like texture or patterns in 2D planes, and 3D posing of these planes in the scene.
Large-Scale Intelligent Microservices
Deploying Machine Learning (ML) algorithms within databases is a challenge due to the varied computational footprints of modern ML algorithms and the myriad of database technologies each with its own restrictive syntax.
Layered Neural Rendering for Retiming People in Video
We present a method for retiming people in an ordinary, natural video -- manipulating and editing the time in which different motions of individuals in the video occur.
Neural Descent for Visual 3D Human Pose and Shape
We present deep neural network methodology to reconstruct the 3d pose and shape of people, given an input RGB image.
Ranked #23 on
3D Human Pose Estimation
on 3DPW
(MPJPE metric)
Neural Light Transport for Relighting and View Synthesis
In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint.
MosAIc: Finding Artistic Connections across Culture with Conditional Image Retrieval
We introduce MosAIc, an interactive web app that allows users to find pairs of semantically related artworks that span different cultures, media, and millennia.
It Is Likely That Your Loss Should be a Likelihood
Joint optimization of these "likelihood parameters" with model parameters can adaptively tune the scales and shapes of losses in addition to the strength of regularization.
Perspective Plane Program Induction from a Single Image
We study the inverse graphics problem of inferring a holistic representation for natural images.
Two-Dimensional Non-Line-of-Sight Scene Estimation from a Single Edge Occluder
Previous work has leveraged the vertical nature of the edge to demonstrate 1D (in angle measured around the corner) reconstructions of moving and stationary hidden scenery from as little as a single photograph of the penumbra.
Deep Audio Priors Emerge From Harmonic Convolutional Networks
We show that networks using Harmonic Convolution can reliably model audio priors and achieve high performance in unsupervised audio restoration tasks.
SpeedNet: Learning the Speediness in Videos
We demonstrate how those learned features can boost the performance of self-supervised action recognition, and can be used for video retrieval.
Semantic Pyramid for Image Generation
We demonstrate that our model results in a versatile and flexible framework that can be used in various classic and novel image generation tasks.
Computational Mirrors: Blind Inverse Light Transport by Deep Matrix Factorization
We recover a video of the motion taking place in a hidden scene by observing changes in indirect illumination in a nearby uncalibrated visible region.
Program-Guided Image Manipulators
Humans are capable of building holistic representations for images at various levels, from local objects, to pairwise relations, to global structures.
Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions
We introduce visual deprojection: the task of recovering an image or video that has been collapsed along a dimension.
Boundless: Generative Adversarial Networks for Image Extension
Image extension models have broad applications in image editing, computational photography and computer graphics.
Ranked #2 on
Uncropping
on Places2 val
Speech2Face: Learning the Face Behind a Voice
How much can we infer about a person's looks from the way they speak?
Learning to Describe Scenes with Programs
We are able to understand the higher-level, abstract regularities within the scene such as symmetry and repetition.
Reasoning About Physical Interactions with Object-Centric Models
Object-based factorizations provide a useful level of abstraction for interacting with the world.
Modeling Parts, Structure, and System Dynamics via Predictive Learning
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future.
Learning the Depths of Moving People by Watching Frozen People
We present a method for predicting dense depth in scenarios where both a monocular camera and people in the scene are freely moving.
Learning Shape Templates with Structured Implicit Functions
To allow for widely varying geometry and topology, we choose an implicit surface representation based on composition of local shape elements.
Visualizing and Understanding GANs
We present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level.
Unsupervised Discovery of Parts, Structure, and Dynamics
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future.
On the Units of GANs (Extended Abstract)
We quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
Learning to Infer and Execute 3D Shape Programs
Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts.
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
Object-based factorizations provide a useful level of abstraction for interacting with the world.
Learning to Reconstruct Shapes from Unseen Classes
From a single image, humans are able to perceive the full 3D shape of an object by exploiting learned shape priors from everyday life.
Visual Object Networks: Image Generation with Disentangled 3D Representation
Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes.
GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
Co-regularized Alignment for Unsupervised Domain Adaptation
Deep neural networks, trained with large amount of labeled data, can fail to generalize well when tested with examples from a \emph{target domain} whose distribution differs from the training data distribution, referred as the \emph{source domain}.
ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Robotics
The underlying physical laws of deformable objects are more complex, and the resulting systems have orders of magnitude more degrees of freedom and therefore they are significantly more computationally expensive to simulate.
MoSculp: Interactive Visualization of Shape and Time
We present a system that allows users to visualize complex human motion via 3D motion sculptures---a representation that conveys the 3D structure swept by a human body as it moves through space.
Learning Shape Priors for Single-View 3D Completion and Reconstruction
The problem of single-view 3D shape completion or reconstruction is challenging, because among the many possible shapes that explain an observation, most are implausible and do not correspond to natural objects.
Seeing Tree Structure from Vibration
Humans recognize object structure from both their appearance and motion; often, motion helps to resolve ambiguities in object structure that arise when we observe object appearance only.
Physical Primitive Decomposition
As annotated data for object parts and physics are rare, we propose a novel formulation that learns physical primitives by explaining both an object's appearance and its behaviors in physical events.
3D-Aware Scene Manipulation via Inverse Graphics
In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model.
Medical Image Imputation from Image Collections
We present an algorithm for creating high resolution anatomically plausible images consistent with acquired clinical brain MRI scans with large inter-slice spacing.
3D Shape Perception from Monocular Vision, Touch, and Shape Priors
Perceiving accurate 3D object shape is important for robots to interact with the physical world.
Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks
We study the problem of synthesizing a number of likely future frames from a single input image.
Unsupervised Training for 3D Morphable Model Regression
We train a regression network using these objectives, a set of unlabeled photographs, and the morphable model itself, and demonstrate state-of-the-art results.
Ranked #2 on
3D Face Reconstruction
on Florence
(Average 3D Error metric)
Learning and Using the Arrow of Time
We seek to understand the arrow of time in videos -- what makes videos look like they are playing forwards or backwards?
Ranked #45 on
Self-Supervised Action Recognition
on UCF101
Self-Supervised Action Recognition
Temporal Action Localization
+1
Sparse, Smart Contours to Represent and Edit Images
We study the problem of reconstructing an image from information stored at contour locations.
Inferring Light Fields From Shadows
We present a method for inferring a 4D light field of a hidden scene from 2D shadows cast by a known occluder on a diffuse wall.
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
We study 3D shape modeling from a single image and make contributions to it in three aspects.
Ranked #1 on
3D Shape Classification
on Pix3D
Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
Solving this task using only audio as input is extremely challenging and does not provide an association of the separated speech signals with speakers in the video.
Learning-based Video Motion Magnification
We show that the learned filters achieve high-quality results on real videos, with less ringing artifacts and better noise characteristics than previous methods.
3D Interpreter Networks for Viewer-Centered Wireframe Modeling
3D-INN is trained on real images to estimate 2D keypoint heatmaps from an input image; it then predicts 3D object structure from heatmaps using knowledge learned from synthetic 3D shapes.
Smart, Sparse Contours to Represent and Edit Images
We study the problem of reconstructing an image from information stored at contour locations.
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
The sound of crashing waves, the roar of fast-moving cars -- sound conveys important information about the objects in our surroundings.
Video Enhancement with Task-Oriented Flow
Many video enhancement algorithms rely on optical flow to register frames in a video sequence.
Ranked #7 on
Video Frame Interpolation
on Middlebury
MarrNet: 3D Shape Reconstruction via 2.5D Sketches
First, compared to full 3D shape, 2. 5D sketches are much easier to be recovered from a 2D image; models that recover 2. 5D sketches are also more likely to transfer from synthetic to real data.
Ranked #2 on
3D Shape Classification
on Pix3D
3D Object Reconstruction From A Single Image
3D Reconstruction
+2
Reconstructing Video from Interferometric Measurements of Time-Varying Sources
Most recently, the Event Horizon Telescope (EHT) has extended VLBI to short millimeter wavelengths with a goal of achieving angular resolution sufficient for imaging the event horizons of nearby supermassive black holes.
Turning Corners Into Cameras: Principles and Methods
We show that walls and other obstructions with edges can be exploited as naturally-occurring "cameras" that reveal the hidden scenes beyond them.
Generative Modeling of Audible Shapes for Object Perception
Humans infer rich knowledge of objects from both auditory and visual cues.
On the Effectiveness of Visible Watermarks
Since such an attack relies on the consistency of watermarks across image collection, we explore and evaluate how it is affected by various types of inconsistencies in the watermark embedding that could potentially be used to make watermarking more secured.
Visual Vibrometry: Estimating Material Properties from Small Motions in Video
The estimation of material properties is important for scene understanding, with many applications in vision, robotics, and structural engineering.
Visual Vibrometry: Estimating MaterialProperties from Small Motions in Video
The estimation of material properties is important for scene understanding, with many applications in vision, robotics, andstructural engineering.
Synthesizing Normalized Faces from Facial Identity Features
We present a method for synthesizing a frontal, neutral-expression image of a person's face given an input face photograph.
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
We study the problem of 3D object generation.
Ranked #3 on
3D Shape Classification
on Pix3D
3D Object Recognition
3D Point Cloud Linear Classification
+1
Best-Buddies Similarity - Robust Template Matching using Mutual Nearest Neighbors
We propose a novel method for template matching in unconstrained environments.
Ambient Sound Provides Supervision for Visual Learning
We show that, through this process, the network learns a representation that conveys information about objects and scenes.
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
We study the problem of synthesizing a number of likely future frames from a single input image.
A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understanding
Humans demonstrate remarkable abilities to predict physical events in complex scenes.
Single Image 3D Interpreter Network
In this work, we propose 3D INterpreter Network (3D-INN), an end-to-end framework which sequentially estimates 2D keypoint heatmaps and 3D object structure, trained on both real 2D-annotated images and synthetic 3D data.
Computational Imaging for VLBI Image Reconstruction
Very long baseline interferometry (VLBI) is a technique for imaging celestial radio emissions by simultaneously observing a source from telescopes distributed across Earth.
Learning Ordinal Relationships for Mid-Level Vision
We demonstrate that this frame- work works well on two important mid-level vision tasks: intrinsic image decomposition and depth from an RGB im- age.
Visual Vibrometry: Estimating Material Properties From Small Motion in Video
The estimation of material properties is important for scene understanding, with many applications in vision, robotics, and structural engineering.
Video Magnification in Presence of Large Motions
Video magnification reveals subtle variations that would be otherwise invisible to the naked eye.
The Aperture Problem for Refractive Motion
We pose and solve a generalization of the aperture problem for moving refractive elements.
Best-Buddies Similarity for Robust Template Matching
We propose a novel method for template matching in unconstrained environments.
Reflection Removal Using Ghosting Cues
For single-pane windows, ghosting cues arise from shifted reflections on the two surfaces of the glass pane.
A Compositional Model for Low-Dimensional Image Set Representation
Learning a low-dimensional representation of images is useful for various applications in graphics and computer vision.
Seeing the Arrow of Time
We explore whether we can observe Time's Arrow in a temporal sequence--is it possible to tell whether a video is running forwards or backwards?
