Search Results for author:
Found 106 papers, 49 papers with code
BoxDreamer: Dreaming Box Corners for Generalizable Object Pose Estimation
This paper presents a generalizable RGB-based approach for object pose estimation, specifically designed to address challenges in sparse-view settings.
AvatarArtist: Open-Domain 4D Avatarization
This work focuses on open-domain 4D avatarization, with the purpose of creating a 4D avatar from a portrait image in an arbitrary style.
Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation
This paper addresses the task of generating two-character online interactions.
FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views
We present FLARE, a feed-forward model designed to infer high-quality camera poses and 3D geometry from uncalibrated sparse-view images (i. e., as few as 2-8 inputs), which is a challenging yet practical setting in real-world applications.
DiffDoctor: Diagnosing Image Diffusion Models Before Treating
In spite of the recent progress, image diffusion models still produce artifacts.
MangaNinja: Line Art Colorization with Precise Reference Following
Derived from diffusion models, MangaNinjia specializes in the task of reference-guided line art colorization.
Edicho: Consistent Image Editing in the Wild
As a verified need, consistent editing across in-the-wild images remains a technical challenge arising from various unmanageable factors, like object poses, lighting conditions, and photography environments.
Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation
In this work, we introduce Prometheus, a 3D-aware latent diffusion model for text-to-3D generation at both object and scene levels in seconds.
DepthLab: From Partial to Complete
Missing values remain a common challenge for depth data across its wide range of applications, stemming from various causes like incomplete data acquisition and perspective alteration.
LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis
The intuitive nature of drag-based interaction has led to its growing adoption for controlling object trajectories in image-to-video synthesis.
EnvGS: Modeling View-Dependent Appearance with Environment Gaussian
Reconstructing complex reflections in real-world scenes from 2D images is essential for achieving photorealistic novel view synthesis.
AniDoc: Animation Creation Made Easier
The production of 2D animation follows an industry-standard workflow, encompassing four essential stages: character design, keyframe animation, in-betweening, and coloring.
Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning
Generating detailed captions comprehending text-rich visual content in images has received growing attention for Large Vision-Language Models (LVLMs).
Learning Visual Generative Priors without Text
Although text-to-image (T2I) models have recently thrived as visual generative priors, their reliance on high-quality text-image pairs makes scaling up expensive.
Mimir: Improving Video Diffusion Models for Precise Text Understanding
Text serves as the key control signal in video generation due to its narrative nature.
PlanarSplatting: Accurate Planar Surface Reconstruction in 3 Minutes
We believe that our accurate and ultrafast planar surface reconstruction method will be applied in the structured data curation for surface reconstruction in the future.
MagicQuill: An Intelligent Interactive Image Editing System
Image editing involves a variety of complex tasks and requires efficient and precise manipulation techniques.
Framer: Interactive Frame Interpolation
We propose Framer for interactive frame interpolation, which targets producing smoothly transitioning frames between two images as per user creativity.
Rectified Diffusion Guidance for Conditional Generation
Classifier-Free Guidance (CFG), which combines the conditional and unconditional score functions with two coefficients summing to one, serves as a practical technique for diffusion model sampling.
LoTLIP: Improving Language-Image Pre-training for Long Text Understanding
Then, after incorporating corner tokens to aggregate diverse textual information, we manage to help the model catch up to its original level of short text understanding yet greatly enhance its capability of long text understanding.
Zero-shot Image Editing with Reference Imitation
Image editing serves as a practical yet challenging task considering the diverse demands from users, where one of the hardest parts is to precisely describe how the edited image should look like.
Learning Temporally Consistent Video Depth from Video Diffusion Priors
Moreover, we design an effective training strategy to provide context within a clip.
GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesis
In this paper, we introduce GaussianPrediction, a novel framework that empowers 3D Gaussian representations with dynamic scene modeling and future scenario synthesis in dynamic environments.
MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
Instead of relying on extensive paired data, i. e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs.
InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior
3D Gaussians have recently emerged as an efficient representation for novel view synthesis.
Learning 3D-Aware GANs from Unposed Images with Template Feature Field
Collecting accurate camera poses of training images has been shown to well serve the learning of 3D-aware generative adversarial networks (GANs) yet can be quite expensive in practice.
SpatialTracker: Tracking Any 2D Pixels in 3D Space
Recovering dense and long-range pixel motion in videos is a challenging problem.
DreamLIP: Language-Image Pre-training with Long Captions
Motivated by this, we propose to dynamically sample sub-captions from the text label to construct multiple positive pairs, and introduce a grouping loss to match the embeddings of each sub-caption with its corresponding local image patches in a self-supervised manner.
FlashFace: Human Image Personalization with High-fidelity Identity Preservation
This work presents FlashFace, a practical tool with which users can easily personalize their own photos on the fly by providing one or a few reference face images and a text prompt.
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation
We introduce GRM, a large-scale reconstructor capable of recovering a 3D asset from sparse-view images in around 0. 1s.
Contextual AD Narration with Interleaved Multimodal Sequence
The Audio Description (AD) task aims to generate descriptions of visual elements for visually impaired individuals to help them access long-form video content, like movies.
Real-time 3D-aware Portrait Editing from a Single Image
Second, thanks to the powerful priors, our module could focus on the learning of editing-related variations, such that it manages to handle various types of editing simultaneously in the training phase and further supports fast adaptation to user-specified customized types of editing during inference (e. g., with ~5min fine-tuning per style).
Towards More Accurate Diffusion Model Acceleration with A Timestep Tuner
A diffusion model which is formulated to produce an image using thousands of denoising steps usually suffers from a slow inference speed.
A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
Following such a pipeline, we study the effect of doubling the scale of training set (i. e., video-only WebVid10M) with some randomly collected text-free videos and are encouraged to observe the performance improvement (FID from 9. 67 to 8. 19 and FVD from 484 to 441), demonstrating the scalability of our approach.
Ranked #7 on
Text-to-Video Generation
on MSR-VTT
TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification
The crux of learning vision-language models is to extract semantically aligned information from visual and linguistic data.
SAM-guided Graph Cut for 3D Instance Segmentation
Experimental results on the ScanNet, ScanNet++ and KITTI-360 datasets demonstrate that our method achieves robust segmentation performance and can generalize across different types of scenes.
CCM: Adding Conditional Controls to Text-to-Image Consistency Models
Consistency Models (CMs) have showed a promise in creating visual content efficiently and with high quality.
HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillation
The noisy image, landmarks, and text condition are then fed into the frozen ControlNet twice for noise prediction.
Learning Naturally Aggregated Appearance for Efficient 3D Editing
Neural radiance fields, which represent a 3D scene as a color field and a density field, have demonstrated great progress in novel view synthesis yet are unfavorable for editing due to the implicitness.
GenDeF: Learning Generative Deformation Field for Video Generation
We offer a new perspective on approaching the task of video generation.
LivePhoto: Real Image Animation with Text-guided Motion Control
In particular, considering the facts that (1) text can only describe motions roughly (e. g., regardless of the moving speed) and (2) text may include both content and motion descriptions, we introduce a motion intensity estimation module as well as a text re-weighting module to reduce the ambiguity of text-to-motion mapping.
BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation
Generating large-scale 3D scenes cannot simply apply existing 3D object synthesis technique since 3D scenes usually hold complex spatial configurations and consist of a number of objects at varying scales.
SMaRt: Improving GANs with Score Matching Regularity
In this work, we revisit the mathematical foundations of GANs, and theoretically reveal that the native adversarial loss for GAN training is insufficient to fix the problem of subsets with positive Lebesgue measure of the generated data manifold lying out of the real data manifold.
4K4D: Real-Time 4D View Synthesis at 4K Resolution
Experiments show that our representation can be rendered at over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the ENeRF-Outdoor dataset at 4K resolution using an RTX 4090 GPU, which is 30x faster than previous methods and achieves the state-of-the-art rendering quality.
Towards More Accurate Diffusion Model Acceleration with A Timestep Aligner
By viewing the generation of diffusion models as a discretized integrating process, we argue that the quality drop is partly caused by applying an inaccurate integral direction to a timestep interval.
In-Domain GAN Inversion for Faithful Reconstruction and Editability
This work fills in this gap by proposing in-domain GAN inversion, which consists of a domain-guided encoder and a domain-regularized optimizer, to regularize the inverted code in the native latent space of the pre-trained GAN model.
Exploring Sparse MoE in GANs for Text-conditioned Image Synthesis
Due to the difficulty in scaling up, generative adversarial networks (GANs) seem to be falling from grace on the task of text-conditioned image synthesis.
CoDeF: Content Deformation Fields for Temporally Consistent Video Processing
With such a design, CoDeF naturally supports lifting image algorithms for video processing, in the sense that one can apply an image algorithm to the canonical image and effortlessly propagate the outcomes to the entire video with the aid of the temporal deformation field.
Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models
To bring the useful knowledge back into light, we first identify a set of parameters that are important to a given downstream task, then attach a binary mask to each parameter, and finally optimize these masks on the downstream data with the parameters frozen.
AnyDoor: Zero-shot Object-level Image Customization
This work presents AnyDoor, a diffusion-based image generator with the power to teleport target objects to new scenes at user-specified locations in a harmonious way.
NEAT: Distilling 3D Wireframes from Neural Attraction Fields
Instead of leveraging matching-based solutions from 2D wireframes (or line segments) for 3D wireframe reconstruction as done in prior arts, we present NEAT, a rendering-distilling formulation using neural fields to represent 3D line segments with 2D observations, and bipartite matching for perceiving and distilling of a sparse set of 3D global junctions.
Lipschitz Singularities in Diffusion Models
Diffusion models, which employ stochastic differential equations to sample images through integrals, have emerged as a dominant class of generative models.
Using Unreliable Pseudo-Labels for Label-Efficient Semantic Segmentation
A common practice is to select the highly confident predictions as the pseudo-ground-truths for each pixel, but it leads to a problem that most pixels may be left unused due to their unreliability.
VideoComposer: Compositional Video Synthesis with Motion Controllability
The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis.
Ranked #5 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
Balancing Logit Variation for Long-tailed Semantic Segmentation
In this way, we manage to close the gap between the feature areas of different categories, resulting in a more balanced representation.
Cones 2: Customizable Image Synthesis with Multiple Subjects
Synthesizing images with user-specified subjects has received growing attention due to its practical applications.
Pulling Target to Source: A New Perspective on Domain Adaptive Semantic Segmentation
To this end, we propose T2S-DA, which we interpret as a form of pulling Target to Source for Domain Adaptation, encouraging the model in learning similar cross-domain features.
Scanning Only Once: An End-to-end Framework for Fast Temporal Grounding in Long Videos
Video temporal grounding aims to pinpoint a video segment that matches the query description.
VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation
A diffusion probabilistic model (DPM), which constructs a forward diffusion process by gradually adding noise to data points and learns the reverse denoising process to generate new samples, has been shown to handle complex data distribution.
Ranked #13 on
Video Generation
on UCF-101
ViM: Vision Middleware for Unified Downstream Transferring
ViM consists of a zoo of lightweight plug-in modules, each of which is independently learned on a midstream dataset with a shared frozen backbone.
Composer: Creative and Controllable Image Synthesis with Composable Conditions
Recent large-scale generative models learned on big data are capable of synthesizing incredible images yet suffer from limited controllability.
UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Training
This work presents a unified knowledge protocol, called UKnow, which facilitates knowledge-based studies from the perspective of data.
Spatial Steerability of GANs via Self-Supervision from Discriminator
In this work, we propose a self-supervised approach to improve the spatial steerability of GANs without searching for steerable directions in the latent space or requiring extra annotations.
Learning 3D-aware Image Synthesis with Unknown Pose Distribution
Existing methods for 3D-aware image synthesis largely depend on the 3D pose distribution pre-estimated on the training set.
GH-Feat: Learning Versatile Generative Hierarchical Features from GANs
In this work we investigate that such a generative feature learned from image synthesis exhibits great potentials in solving a wide range of computer vision tasks, including both generative ones and more importantly discriminative ones.
LinkGAN: Linking GAN Latents to Pixels for Controllable Image Synthesis
This work presents an easy-to-use regularizer for GAN training, which helps explicitly link some axes of the latent space to a set of pixels in the synthesized image.
LipFormer: High-Fidelity and Generalizable Talking Face Generation With a Pre-Learned Facial Codebook
Generating a talking face video from the input audio sequence is a practical yet challenging task.
DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis
Existing 3D-aware image synthesis approaches mainly focus on generating a single canonical object and show limited capacity in composing a complex scene containing a variety of objects.
Towards Smooth Video Composition
Video generation requires synthesizing consistent and persistent frames with dynamic content over time.
Ranked #1 on
Video Generation
on YouTube Driving
GLeaD: Improving GANs with A Generator-Leading Task
Generative adversarial network (GAN) is formulated as a two-player game between a generator (G) and a discriminator (D), where D is asked to differentiate whether an image comes from real data or is produced by G. Under such a formulation, D plays as the rule maker and hence tends to dominate the competition.
Dimensionality-Varying Diffusion Process
Diffusion models, which learn to reverse a signal destruction process to generate new data, typically require the signal at each step to have the same dimension.
Neural Dependencies Emerging from Learning Massive Categories
Through investigating the properties of the problem solution, we confirm that neural dependency is guaranteed by a redundant logit covariance matrix, which condition is easily met given massive categories, and that neural dependency is highly sparse, implying that one category correlates to only a few others.
Deep Generative Models on 3D Representations: A Survey
In this survey, we thoroughly review the ongoing developments of 3D generative models, including methods that employ 2D and 3D supervision.
Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
We argue that, considering the two-player game in the formulation of GANs, only making the generator 3D-aware is not enough.
Improving GANs with A Dynamic Discriminator
Two capacity adjusting schemes are developed for training GANs under different data regimes: i) given a sufficient amount of training data, the discriminator benefits from a progressively increased learning capacity, and ii) when the training data is limited, gradually decreasing the layer width mitigates the over-fitting issue of the discriminator.
Learning from Future: A Novel Self-Training Framework for Semantic Segmentation
Self-training has shown great potential in semi-supervised learning.
A Unified Model for Multi-class Anomaly Detection
For example, when learning a unified model for 15 categories in MVTec-AD, we surpass the second competitor on the tasks of both anomaly detection (from 88. 1% to 96. 5%) and anomaly localization (from 89. 5% to 96. 8%).
Ranked #3 on
Multi-class Anomaly Detection
on MVTec AD
Interpreting Class Conditional GANs with Channel Awareness
To describe such a phenomenon, we propose channel awareness, which quantitatively characterizes how a single channel contributes to the final synthesis.
High-fidelity GAN Inversion with Padding Space
In this work, we propose to involve the padding space of the generator to complement the latent space with spatial information.
Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels
A common practice is to select the highly confident predictions as the pseudo ground-truth, but it leads to a problem that most pixels may be left unused due to their unreliability.
Region-Based Semantic Factorization in GANs
Despite the rapid advancement of semantic discovery in the latent space of Generative Adversarial Networks (GANs), existing approaches either are limited to finding global attributes or rely on a number of segmentation masks to identify local attributes.
3D-Aware Indoor Scene Synthesis with Depth Priors
In this way, the discriminator can take the spatial arrangement into account and advise the generator to learn an appropriate depth condition.
3D-aware Image Synthesis via Learning Structural and Textural Representations
The feature field is further accumulated into a 2D feature map as the textural representation, followed by a neural renderer for appearance synthesis.
Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognition
Typically in recent work, the pseudo-labels are obtained by training a model on the labeled data, and then using confident predictions from the model to teach itself.
Improving GAN Equilibrium by Raising Spatial Awareness
We further propose to align the spatial awareness of G with the attention map induced from D. Through this way we effectively lessen the information gap between D and G. Extensive results show that our method pushes the two-player game in GANs closer to the equilibrium, leading to a better synthesis performance.
CompConv: A Compact Convolution Module for Efficient Feature Learning
Convolutional Neural Networks (CNNs) have achieved remarkable success in various computer vision tasks but rely on tremendous computational cost.
Glancing at the Patch: Anomaly Localization With Global and Local Feature Comparison
More concretely, we employ a Local-Net and Global-Net to extract features from any individual patch and its surrounding respectively.
Data-Efficient Instance Generation from Instance Discrimination
Meanwhile, the learned instance discrimination capability from the discriminator is in turn exploited to encourage the generator for diverse generation.
Ranked #11 on
Image Generation
on FFHQ 256 x 256
(FD metric)
Low-Rank Subspaces in GANs
Concretely, given an arbitrary image and a region of interest (e. g., eyes of face images), we manage to relate the latent space to the image region with the Jacobian matrix and then use low-rank factorization to discover steerable latent subspaces.
Decorating Your Own Bedroom: Locally Controlling Image Generation with Generative Adversarial Networks
Generative Adversarial Networks (GANs) have made great success in synthesizing high-quality images.
Unsupervised Image Transformation Learning via Generative Adversarial Networks
In this work, we study the image transformation problem, which targets at learning the underlying transformations (e. g., the transition of seasons) from a collection of unlabeled images.
Generative Hierarchical Features from Synthesizing Images
Generative Adversarial Networks (GANs) have recently advanced image synthesis by learning the underlying distribution of the observed data.
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs
In this work, we propose a framework called InterFaceGAN to interpret the disentangled face representation learned by the state-of-the-art GAN models and study the properties of the facial semantics encoded in the latent space.
In-Domain GAN Inversion for Real Image Editing
A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code.
Residual Knowledge Distillation
Knowledge distillation (KD) is one of the most potent ways for model compression.
Image Processing Using Multi-Code GAN Prior
Such an over-parameterization of the latent space significantly improves the image reconstruction quality, outperforming existing competitors.
Ranked #8 on
Blind Face Restoration
on CelebA-Test
Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what generative models have learned inside the deep generative representations and how photo-realistic images are able to be composed of the layer-wise stochasticity introduced in recent GANs.
Semantic Hierarchy Emerges in the Deep Generative Representations for Scene Synthesis
Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises.
Interpreting the Latent Space of GANs for Semantic Face Editing
In this work, we propose a novel framework, called InterFaceGAN, for semantic face editing by interpreting the latent semantics learned by GANs.
An Embarrassingly Simple Approach for Knowledge Distillation
Knowledge Distillation (KD) aims at improving the performance of a low-capacity student model by inheriting knowledge from a high-capacity teacher model.
FaceFeat-GAN: a Two-Stage Approach for Identity-Preserving Face Synthesis
In the second stage, they compete in the image domain to render photo-realistic images that contain high diversity but preserve identity.
FaceID-GAN: Learning a Symmetry Three-Player GAN for Identity-Preserving Face Synthesis
Existing methods typically formulate GAN as a two-player game, where a discriminator distinguishes face images from the real and synthesized domains, while a generator reduces its discriminativeness by synthesizing a face of photo-realistic quality.
