Search Results for author:
Found 95 papers, 60 papers with code
Lazy Diffusion Transformer for Interactive Image Editing
We demonstrate that our approach is competitive with state-of-the-art inpainting methods in terms of quality and fidelity while providing a 10x speedup for typical user interactions, where the editing mask represents 10% of the image.
Customizing Text-to-Image Diffusion with Camera Viewpoint Control
Model customization introduces new concepts to existing text-to-image models, enabling the generation of the new concept in novel contexts.
VideoGigaGAN: Towards Detail-rich Video Super-Resolution
We introduce VideoGigaGAN, a new generative VSR model that can produce videos with high-frequency details and temporal consistency.
Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos
We show that by using simple segmentations and coarse 2D manipulations, we can synthesize a photorealistic edit faithful to the user's input while addressing second-order effects like harmonizing the lighting and physical interactions between edited objects.
Jump Cut Smoothing for Talking Heads
We leverage the appearance of the subject from the other source frames in the video, fusing it with a mid-level representation driven by DensePose keypoints and face landmarks.
Customizing Motion in Text-to-Video Diffusion Models
To avoid overfitting to the new custom motion, we introduce an approach for regularization over videos.
One-step Diffusion with Distribution Matching Distillation
We introduce Distribution Matching Distillation (DMD), a procedure to transform a diffusion model into a one-step image generator with minimal impact on image quality.
Perceptual Artifacts Localization for Image Synthesis Tasks
Recent advancements in deep generative models have facilitated the creation of photo-realistic images across various tasks.
DIFF-NST: Diffusion Interleaving For deFormable Neural Style Transfer
Neural Style Transfer (NST) is the field of study applying neural techniques to modify the artistic appearance of a content image to match the style of a reference style image.
Realistic Saliency Guided Image Enhancement
We thus offer a viable solution for automating image enhancement and photo cleanup operations.
SimpSON: Simplifying Photo Cleanup with Single-Click Distracting Object Segmentation Network
In photo editing, it is common practice to remove visual distractions to improve the overall image quality and highlight the primary subject.
NeAT: Neural Artistic Tracing for Beautiful Style Transfer
As a component of curating this data, we present a novel model able to classify if an image is stylistic.
Automatic High Resolution Wire Segmentation and Removal
In this paper, we present an automatic wire clean-up system that eases the process of wire segmentation and removal/inpainting to within a few seconds.
Ablating Concepts in Text-to-Image Diffusion Models
To achieve this goal, we propose an efficient method of ablating concepts in the pretrained model, i. e., preventing the generation of a target concept.
Scaling up GANs for Text-to-Image Synthesis
From a technical standpoint, it also marked a drastic change in the favored architecture to design generative image models.
Ranked #18 on
Image Generation
on ImageNet 256x256
Semi-supervised Parametric Real-world Image Harmonization
Learning-based image harmonization techniques are usually trained to undo synthetic random global transformations applied to a masked foreground in a single ground truth photo.
Domain Expansion of Image Generators
First, we note the generator contains a meaningful, pretrained latent space.
Structure-Guided Image Completion with Image-level and Object-level Semantic Discriminators
Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects.
Multi-Concept Customization of Text-to-Image Diffusion
Can we teach a model to quickly acquire a new concept, given a few examples?
Contrastive Learning for Diverse Disentangled Foreground Generation
We introduce a new method for diverse foreground generation with explicit control over various factors.
Text-Free Learning of a Natural Language Interface for Pretrained Face Generators
We propose Fast text2StyleGAN, a natural language interface that adapts pre-trained GANs for text-guided human face synthesis.
Inpainting at Modern Camera Resolution by Guided PatchMatch with Auto-Curation
Recently, deep models have established SOTA performance for low-resolution image inpainting, but they lack fidelity at resolutions associated with modern cameras such as 4K or more, and for large holes.
Perceptual Artifacts Localization for Inpainting
Inspired by this workflow, we propose a new learning task of automatic segmentation of inpainting perceptual artifacts, and apply the model for inpainting model evaluation and iterative refinement.
Controllable Shadow Generation Using Pixel Height Maps
It can be used to calculate hard shadows in a 2D image based on the projective geometry, providing precise control of the shadows' direction and shape.
RigNeRF: Fully Controllable Neural 3D Portraits
In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video.
ARF: Artistic Radiance Fields
We present a method for transferring the artistic features of an arbitrary style image to a 3D scene.
BlobGAN: Spatially Disentangled Scene Representations
Blobs are differentiably placed onto a feature grid that is decoded into an image by a generative adversarial network.
Any-resolution Training for High-resolution Image Synthesis
To take advantage of varied-size data, we introduce continuous-scale training, a process that samples patches at random scales to train a new generator with variable output resolutions.
Neural Neighbor Style Transfer
We propose Neural Neighbor Style Transfer (NNST), a pipeline that offers state-of-the-art quality, generalization, and competitive efficiency for artistic style transfer.
CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training
We propose cascaded modulation GAN (CM-GAN), a new network design consisting of an encoder with Fourier convolution blocks that extract multi-scale feature representations from the input image with holes and a dual-stream decoder with a novel cascaded global-spatial modulation block at each scale level.
Ranked #1 on
Image Inpainting
on Places2
InsetGAN for Full-Body Image Generation
Instead of modeling this complex domain with a single GAN, we propose a novel method to combine multiple pretrained GANs, where one GAN generates a global canvas (e. g., human body) and a set of specialized GANs, or insets, focus on different parts (e. g., faces, shoes) that can be seamlessly inserted onto the global canvas.
Third Time's the Charm? Image and Video Editing with StyleGAN3
In particular, we demonstrate that while StyleGAN3 can be trained on unaligned data, one can still use aligned data for training, without hindering the ability to generate unaligned imagery.
GeoFill: Reference-Based Image Inpainting with Better Geometric Understanding
Our approach achieves state-of-the-art performance on both RealEstate10K and MannequinChallenge dataset with large baselines, complex geometry and extreme camera motions.
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation
We introduce a high resolution, 3D-consistent image and shape generation technique which we call StyleSDF.
Ensembling Off-the-shelf Models for GAN Training
Can the collective "knowledge" from a large bank of pretrained vision models be leveraged to improve GAN training?
Ranked #1 on
Image Generation
on AFHQ Cat
GAN-Supervised Dense Visual Alignment
We propose GAN-Supervised Learning, a framework for learning discriminative models and their GAN-generated training data jointly end-to-end.
StyleAlign: Analysis and Applications of Aligned StyleGAN Models
Several works already utilize some basic properties of aligned StyleGAN models to perform image-to-image translation.
STALP: Style Transfer with Auxiliary Limited Pairing
We present an approach to example-based stylization of images that uses a single pair of a source image and its stylized counterpart.
Real Image Inversion via Segments
In this short report, we present a simple, yet effective approach to editing real images via generative adversarial networks (GAN).
KDSalBox: A toolbox of efficient knowledge-distilled saliency models
Dozens of saliency models have been designed over the last few decades, targeted at diverse applications ranging from image compression and retargeting to robot navigation, surveillance, and distractor detection.
Collaging Class-specific GANs for Semantic Image Synthesis
We propose a new approach for high resolution semantic image synthesis.
Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN
We present an algorithm for re-rendering a person from a single image under arbitrary poses.
Ensembling with Deep Generative Views
Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification.
Few-shot Image Generation via Cross-domain Correspondence
Training generative models, such as GANs, on a target domain containing limited examples (e. g., 10) can easily result in overfitting.
Ranked #3 on
10-shot image generation
on Babies
Modulated Periodic Activations for Generalizable Local Functional Representations
Our approach produces generalizable functional representations of images, videos and shapes, and achieves higher reconstruction quality than prior works that are optimized for a single signal.
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Inspired by the ability of StyleGAN to generate highly realistic images in a variety of domains, much recent work has focused on understanding how to use the latent spaces of StyleGAN to manipulate generated and real images.
TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations
Image inpainting is the task of plausibly restoring missing pixels within a hole region that is to be removed from a target image.
CharacterGAN: Few-Shot Keypoint Character Animation and Reposing
Our model generates novel poses based on keypoint locations, which can be modified in real time while providing interactive feedback, allowing for intuitive reposing and animation.
Spatially-Adaptive Pixelwise Networks for Fast Image Translation
We introduce a new generator architecture, aimed at fast and efficient high-resolution image-to-image translation.
Few-shot Image Generation with Elastic Weight Consolidation
Few-shot image generation seeks to generate more data of a given domain, with only few available training examples.
Ranked #4 on
10-shot image generation
on Babies
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation
Manipulation of visual attributes via these StyleSpace controls is shown to be better disentangled than via those proposed in previous works.
Look here! A parametric learning based approach to redirect visual attention
Extensions of our model allow for multi-style edits and the ability to both increase and attenuate attention in an image region.
Swapping Autoencoder for Deep Image Manipulation
Deep generative models have become increasingly effective at producing realistic images from randomly sampled seeds, but using such models for controllable manipulation of existing images remains challenging.
High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling
To address this challenge, we propose an iterative inpainting method with a feedback mechanism.
Ranked #6 on
Image Inpainting
on Places2
Image Morphing with Perceptual Constraints and STN Alignment
In image morphing, a sequence of plausible frames are synthesized and composited together to form a smooth transformation between given instances.
MakeItTalk: Speaker-Aware Talking-Head Animation
We present a method that generates expressive talking heads from a single facial image with audio as the only input.
State of the Art on Neural Rendering
Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e. g., by the integration of differentiable rendering into network training.
Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement
We present a method to improve the visual realism of low-quality, synthetic images, e. g. OpenGL renderings.
Lifespan Age Transformation Synthesis
Most existing aging methods are limited to changing the texture, overlooking transformations in head shape that occur during the human aging and growth process.
Neural Puppet: Generative Layered Cartoon Characters
We capture these subtle changes by applying an image translation network to refine the mesh rendering, providing an end-to-end model to generate new animations of a character with high visual quality.
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
We propose an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects.
UprightNet: Geometry-Aware Camera Orientation Estimation from Single Images
We introduce UprightNet, a learning-based approach for estimating 2DoF camera orientation from a single RGB image of an indoor scene.
Text-based Editing of Talking-head Video
To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material.
Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction
In this paper, we address the problem of 3D object mesh reconstruction from RGB videos.
Im2Pencil: Controllable Pencil Illustration from Photographs
We propose a high-quality photo-to-pencil translation method with fine-grained control over the drawing style.
Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture
This paper addresses the problem of interpolating visual textures.
Localizing Moments in Video with Temporal Language
To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset.
MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
Our model jointly learns a feature embedding for motion modes (that the motion sequence can be reconstructed from) and a feature transformation that represents the transition of one motion mode to the next motion mode.
Ranked #7 on
Human Pose Forecasting
on Human3.6M
(ADE metric)
Learning Blind Video Temporal Consistency
Our method takes the original unprocessed and per-frame processed videos as inputs to produce a temporally consistent video.
Deep Painterly Harmonization
Copying an element from a photo and pasting it into a painting is a challenging task.
Graphics
ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
We address the problem of finding realistic geometric corrections to a foreground object such that it appears natural when composited into a background image.
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics.
Ranked #19 on
Video Quality Assessment
on MSU FR VQA Database
Multi-Content GAN for Few-Shot Font Style Transfer
In this work, we focus on the challenge of taking partial observations of highly-stylized text and generalizing the observations to generate unobserved glyphs in the ornamented typeface.
Toward Multimodal Image-to-Image Translation
Our proposed method encourages bijective consistency between the latent encoding and output modes.
Photorealistic Style Transfer with Screened Poisson Equation
Recent work has shown impressive success in transferring painterly style to images.
Training Deep Networks to be Spatially Sensitive
In many computer vision tasks, for example saliency prediction or semantic segmentation, the desired output is a foreground map that predicts pixels where some criteria is satisfied.
Localizing Moments in Video with Natural Language
A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment.
Neural Face Editing with Intrinsic Image Disentangling
Traditional face editing methods often require a number of sophisticated and task specific algorithms to be applied one after the other --- a process that is tedious, fragile, and computationally intensive.
Deep Photo Style Transfer
This paper introduces a deep-learning approach to photographic style transfer that handles a large variety of image content while faithfully transferring the reference style.
Removing Shadows from Images of Documents
In this work, we automatically detect and remove distracting shadows from photographs of documents and other text-based items.
Saliency Driven Image Manipulation
Have you ever taken a picture only to find out that an unimportant background object ended up being overly salient?
High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis
Recent advances in deep learning have shown exciting promise in filling large holes in natural images with semantically plausible and context aware details, impacting fundamental image manipulation tasks such as object removal.
Controlling Perceptual Factors in Neural Style Transfer
Neural Style Transfer has shown very exciting results enabling new forms of image manipulation.
Generative Visual Manipulation on the Natural Image Manifold
Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result.
Preserving Color in Neural Artistic Style Transfer
This note presents an extension to the neural artistic style transfer algorithm (Gatys et al.).
Appearance Harmonization for Single Image Shadow Removal
In this work we propose a fully automatic shadow region harmonization approach that improves the appearance compatibility of the de-shadowed region as typically produced by previous methods.
PatchMatch-Based Automatic Lattice Detection for Near-Regular Textures
In this work, we investigate the problem of automatically inferring the lattice structure of near-regular textures (NRT) in real-world images.
Learning a Discriminative Model for the Perception of Realism in Composite Images
What makes an image appear realistic?
DeepFont: Identify Your Font from An Image
As font is one of the core design concepts, automatic font identification and similar font suggestion from an image or photo has been on the wish list of many designers.
Ranked #1 on
Font Recognition
on VFR-Wild
Finding Distractors In Images
We propose a new computer vision task we call "distractor prediction."
Real-World Font Recognition Using Deep Network and Domain Adaptation
We address a challenging fine-grain classification problem: recognizing a font style from an image of text.
Decomposition-Based Domain Adaptation for Real-World Font Recognition
We present a domain adaption framework to address a domain mismatch between synthetic training and real-world testing data.
Large-Scale Visual Font Recognition
This paper addresses the large-scale visual font recognition (VFR) problem, which aims at automatic identification of the typeface, weight, and slope of the text in an image or photo without any knowledge of content.
Ranked #1 on
Font Recognition
on VFR-447
Learning Video Saliency from Human Gaze Using Candidate Selection
For example, the time each video frame is observed is a fraction of a second, while a still image can be viewed leisurely.
Crowdsourcing Gaze Data Collection
In this work we propose a crowdsourced method for acquisition of gaze direction data from a virtually unlimited number of participants, using a robust self-reporting mechanism (see Figure 1).
Social and Information Networks Human-Computer Interaction
