Search Results for author:
Found 42 papers, 33 papers with code
SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation
We present SadTalker, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation.
VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild
Our system disentangles this objective into three sequential tasks: (1) face video generation with a canonical expression; (2) audio-driven lip-sync; and (3) face enhancement for improving photo-realism.
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
The I2V model is designed to produce videos that strictly adhere to the content of the provided reference image, preserving its content, structure, and style.
Ranked #3 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
Based on this stronger coupling, we shift the distribution to higher quality without motion degradation by finetuning spatial modules with high-quality images, resulting in a generic high-quality video model.
Ranked #1 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing
We also have a better zero-shot shape-aware editing ability based on the text-to-video model.
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos
Generating text-editable and pose-controllable character videos have an imperious demand in creating various digital human.
Uformer: A General U-Shaped Transformer for Image Restoration
Powered by these two designs, Uformer enjoys a high capability for capturing both local and global dependencies for image restoration.
Ranked #2 on
Deblurring
on RealBlur-R (trained on GoPro)
StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN
Our framework elevates the resolution of the synthesized talking face to 1024*1024 for the first time, even though the training dataset has a lower resolution.
T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations
Additionally, we conduct analyses on HumanML3D and observe that the dataset size is a limitation of our approach.
Ranked #10 on
Motion Synthesis
on HumanML3D
CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior
In this paper, we propose to cast speech-driven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty.
Ranked #4 on
3D Face Animation
on BEAT2
ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models
Our work also suggests that a pre-trained diffusion model trained on low-resolution images can be directly used for high-resolution visual generation without further tuning, which may provide insights for future research on ultra-high-resolution image and video synthesis.
DPE: Disentanglement of Pose and Expression for General Video Portrait Editing
In this paper, we introduce a novel self-supervised disentanglement framework to decouple pose and expression without 3DMMs and paired data, which consists of a motion editing module, a pose generator, and an expression generator.
AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
For appearance control, we borrow intermediate latents and their features from the text-to-image (T2I) generation for ensuring the generated first frame is equal to the given generated image.
Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN
With the help of novel masks or scenes, we enhance the current datasets using synthesized shadow images.
Ranked #2 on
Shadow Removal
on ISTD
Inserting Anybody in Diffusion Models via Celeb Basis
Empowered by the proposed celeb basis, the new identity in our customized model showcases a better concept combination ability than previous personalization methods.
TaleCrafter: Interactive Story Visualization with Multiple Characters
Accurate Story visualization requires several necessary elements, such as identity consistency across frames, the alignment between plain text and visual content, and a reasonable layout of objects in images.
Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
For the first module, we leverage an off-the-shelf video retrieval system and extract video depths as motion structure.
Split then Refine: Stacked Attention-guided ResUNets for Blind Single Image Visible Watermark Removal
Simultaneously, to increase the robustness of watermark, attacking technique, such as watermark removal, also gets the attention from the community.
Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Framework
We adopt a two-stage training strategy for the diffusion model, effectively binding movements with specific appearances.
High-Resolution Document Shadow Removal via A Large-Scale Real-World Dataset and A Frequency-Aware Shadow Erasing Net
We handle high-resolution document shadow removal directly via a larger-scale real-world dataset and a carefully designed frequency-aware network.
Sketch Video Synthesis
Understanding semantic intricacies and high-level concepts is essential in image sketch generation, and this challenge becomes even more formidable when applied to the domain of videos.
Explicit Visual Prompting for Low-Level Structure Segmentations
Different from the previous visual prompting which is typically a dataset-level implicit embedding, our key insight is to enforce the tunable parameters focusing on the explicit visual content from each individual image, i. e., the features from frozen patch embeddings and the input's high-frequency components.
Ranked #1 on
Salient Object Detection
on DUT-OMRON
Explicit Visual Prompting for Universal Foreground Segmentations
We take inspiration from the widely-used pre-training and then prompt tuning protocols in NLP and propose a new visual prompting model, named Explicit Visual Prompting (EVP).
Ranked #1 on
Salient Object Detection
on HKU-IS
SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models
Both quantitative and qualitative results on this evaluation dataset indicate that our SmartEdit surpasses previous methods, paving the way for the practical application of complex instruction-based image editing.
CoordFill: Efficient High-Resolution Image Inpainting via Parameterized Coordinate Querying
Thanks to the proposed structure, we only encode the high-resolution image in a relatively low resolution for larger reception field capturing.
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
For video generation, various open-sourced models and public-available services have been developed to generate high-quality videos.
MagicStick: Controllable Video Editing via Control Handle Transformations
Yet succinct, our method is the first method to show the ability of video property editing from the pre-trained text-to-image model.
LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation
While previous methods are able to generate speech rhythm-synchronized gestures, the semantic context of the speech is generally lacking in the gesticulations.
Defocus Blur Detection via Depth Distillation
In detail, we learn the defocus blur from ground truth and the depth distilled from a well-trained depth estimation network at the same time.
Improving the Harmony of the Composite Image by Spatial-Separated Attention Module
Thus, we address the problem of Image Harmonization: Given a spliced image and the mask of the spliced region, we try to harmonize the "style" of the pasted region with the background (non-spliced region).
Ranked #5 on
Image Harmonization
on HAdobe5k(1024$\times$1024)
Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation
Diffusion models have proven to be highly effective in image and video generation; however, they still face composition challenges when generating images of varying sizes due to single-scale training data.
Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization
To this end, we propose a novel spatial-separated curve rendering network(S$^2$CRNet) for efficient and high-resolution image harmonization for the first time.
Ranked #12 on
Image Harmonization
on iHarmony4
ShaDocNet: Learning Spatial-Aware Tokens in Transformer for Document Shadow Removal
Shadow removal improves the visual quality and legibility of digital copies of documents.
Depth Assisted Full Resolution Network for Single Image-based View Synthesis
In this paper, we focus on a more challenging and ill-posed problem that is to synthesize novel viewpoints from one single input image.
Learning Enriched Illuminants for Cross and Single Sensor Color Constancy
Color constancy aims to restore the constant colors of a scene under different illuminants.
3D GAN Inversion with Facial Symmetry Prior
It is natural to associate 3D GANs with GAN inversion methods to project a real image into the generator's latent space, allowing free-view consistent synthesis and editing, referred as 3D GAN inversion.
Generating Human Motion From Textual Descriptions With Discrete Representations
Additionally, we conduct analyses on HumanML3D and observe that the dataset size is a limitation of our approach.
Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance
Our method, dubbed Make-Your-Video, involves joint-conditional video generation using a Latent Diffusion Model that is pre-trained for still image synthesis and then promoted for video generation with the introduction of temporal modules.
ToonTalker: Cross-Domain Face Reenactment
Moreover, since no paired data is provided, we propose a novel cross-domain training scheme using data from two domains with the designed analogy constraint.
X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model
To enhance the guidance ability of X-Adapter, we employ a null-text training strategy for the upgraded model.
Towards A Better Metric for Text-to-Video Generation
Experiments on the TVGE dataset demonstrate the superiority of the proposed T2VScore on offering a better metric for text-to-video generation.
Depth-aware Test-Time Training for Zero-shot Video Object Segmentation
In this work, we introduce a test-time training (TTT) strategy to address the problem.
