Search Results for author:
Found 43 papers, 21 papers with code
DreamRelation: Relation-Centric Video Customization
First, in Relational Decoupling Learning, we disentangle relations from subject appearances using relation LoRA triplet and hybrid mask training strategy, ensuring better generalization across diverse relationships.
FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusion
Visual diffusion models achieve remarkable progress, yet they are typically trained at limited resolutions due to the lack of high-resolution data and constrained computation resources, hampering their ability to generate high-fidelity images or videos at higher resolutions.
Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
As a fundamental backbone for video generation, diffusion models are challenged by low inference speed due to the sequential nature of denoising.
PersonalVideo: High ID-Fidelity Video Customization without Dynamic and Semantic Degradation
The key challenge lies in maintaining high ID fidelity consistently while preserving the original motion dynamic and semantic following after the identity injection.
EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
Video generation has emerged as a promising tool for world simulation, leveraging visual data to replicate real-world environments.
DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Control
In this paper, we present DreamVideo-2, a zero-shot video customization framework capable of generating videos with a specific subject and motion trajectory, guided by a single image and a bounding box sequence, respectively, and without the need for test-time fine-tuning.
EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models
Our experiments with extensive data indicate that the model trained on generated data of the advanced model can approximate its generation capability.
FreeMask: Rethinking the Importance of Attention Masks for Zero-Shot Video Editing
Text-to-video diffusion models have made remarkable advancements.
S^3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis
Talking head synthesis is a practical technique with wide applications.
UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation
First, to reduce the optimization difficulty and ensure temporal coherence, we map the reference image along with the posture guidance and noise video into a common feature space by incorporating a unified video diffusion model.
A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
Following such a pipeline, we study the effect of doubling the scale of training set (i. e., video-only WebVid10M) with some randomly collected text-free videos and are encouraged to observe the performance improvement (FID from 9. 67 to 8. 19 and FVD from 484 to 441), demonstrating the scalability of our approach.
Ranked #7 on
Text-to-Video Generation
on MSR-VTT
InstructVideo: Instructing Video Diffusion Models with Human Feedback
To tackle this problem, we propose InstructVideo to instruct text-to-video diffusion models with human feedback by reward fine-tuning.
AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
Audio-driven talking head synthesis is a promising topic with wide applications in digital human, film making and virtual reality.
DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models
To more conveniently specify personalized emotions, a diffusion-based style predictor is utilized to predict the personalized emotion directly from the audio, eliminating the need for extra emotion reference.
VideoLCM: Video Latent Consistency Model
Consistency models have demonstrated powerful capability in efficient image generation and allowed synthesis within a few sampling steps, alleviating the high computational cost in diffusion models.
DreamVideo: Composing Your Dream Videos with Customized Subject and Motion
In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern.
Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation
At the structure level, we decompose the T2V task into two steps, including spatial reasoning and temporal reasoning, using a unified denoiser.
Ranked #6 on
Text-to-Video Generation
on MSR-VTT
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models
By this means, I2VGen-XL can simultaneously enhance the semantic accuracy, continuity of details and clarity of generated videos.
Few-shot Action Recognition with Captioning Foundation Models
In this paper, we develop an effective plug-and-play framework called CapFSAR to exploit the knowledge of multimodal models without manually annotating text.
DeltaSpace: A Semantic-aligned Feature Space for Flexible Text-guided Image Editing
Based on DeltaSpace, we propose a novel framework called DeltaEdit, which maps the CLIP visual feature differences to the latent space directions of a generative model during the training phase, and predicts the latent space directions from the CLIP textual feature differences during the inference phase.
Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning
When pre-training on the large-scale Kinetics-710, we achieve 89. 7% on Kinetics-400 with a frozen ViT-L model, which verifies the scalability of DiST.
RLIPv2: Fast Scaling of Relational Language-Image Pre-training
In this paper, we propose RLIPv2, a fast converging model that enables the scaling of relational pre-training to large-scale pseudo-labelled scene graph data.
Ranked #1 on
Zero-Shot Human-Object Interaction Detection
on HICO-DET
(using extra training data)
ModelScope Text-to-Video Technical Report
This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i. e., Stable Diffusion).
Ranked #9 on
Text-to-Video Generation
on MSR-VTT
Temporally-Adaptive Models for Efficient Video Understanding
Spatial convolutions are extensively used in numerous deep video models.
Ranked #4 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
CMDFusion: Bidirectional Fusion Network with Cross-modality Knowledge Distillation for LIDAR Semantic Segmentation
In this way, RGB images are not required during inference anymore since the 2D knowledge branch provides 2D information according to the 3D LIDAR input.
Freestyle 3D-Aware Portrait Synthesis Based on Compositional Generative Priors
Efficiently generating a freestyle 3D portrait with high quality and 3D-consistency is a promising yet challenging task.
VideoComposer: Compositional Video Synthesis with Motion Controllability
The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis.
Ranked #5 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognition
To address these issues, we develop a Motion-augmented Long-short Contrastive Learning (MoLo) method that contains two crucial components, including a long-short contrastive objective and a motion autodecoder.
Enlarging Instance-specific and Class-specific Information for Open-set Action Recognition
In this paper, we begin with analyzing the feature representation behavior in the open-set action recognition (OSAR) problem based on the information bottleneck (IB) theory, and propose to enlarge the instance-specific (IS) and class-specific (CS) information contained in the feature for better performance.
VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation
A diffusion probabilistic model (DPM), which constructs a forward diffusion process by gradually adding noise to data points and learns the reverse denoising process to generate new samples, has been shown to handle complex data distribution.
Ranked #11 on
Video Generation
on UCF-101
CLIP-guided Prototype Modulating for Few-shot Action Recognition
Learning from large-scale contrastive language-image pre-training like CLIP has shown remarkable success in a wide range of downstream tasks recently, but it is still under-explored on the challenging few-shot action recognition (FSAR) task.
The Devil is in the Wrongly-classified Samples: Towards Unified Open-set Recognition
Recently, Unified Open-set Recognition (UOSR) has been proposed to reject not only unknown samples but also known but wrongly classified samples, which tends to be more practical in real-world applications.
Space-time Prompting for Video Class-incremental Learning
In this paper, we will fill this gap by learning multiple prompts based on a powerful image-language pre-trained model, i. e., CLIP, making it fit for video class-incremental learning (VCIL).
LipFormer: High-Fidelity and Generalizable Talking Face Generation With a Pre-Learned Facial Codebook
Generating a talking face video from the input audio sequence is a practical yet challenging task.
Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimization
While (Lu and Sa, 2021) have recently provided an optimal rate for non-convex stochastic decentralized optimization with weight matrices defined over linear graphs, the optimal rate with general weight matrices remains unclear.
Communicate Then Adapt: An Effective Decentralized Adaptive Method for Deep Training
Decentralized adaptive gradient methods, in which each node averages only with its neighbors, are critical to save communication and wall-clock training time in deep learning tasks.
Communication Efficient SGD via Gradient Sampling With Bayes Prior
Different from all of them, we regard large and small gradients selection as the exploitation and exploration of gradient information, respectively.
Accelerating Gossip SGD with Periodic Global Averaging
Communication overhead hinders the scalability of large-scale distributed training.
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training
Experimental results on a variety of computer vision tasks and models demonstrate that DecentLaM promises both efficient and high-quality training.
Visual Search at Alibaba
We hope visual search at Alibaba becomes more widely incorporated into today's commercial applications.
Large-Scale Visual Search with Binary Distributed Graph at Alibaba
For a deployed visual search system with several billions of online images in total, building a billion-scale offline graph in hours is essential, which is almost unachievable by most existing methods.
Distribution Adaptive INT8 Quantization for Training CNNs
Researches have demonstrated that low bit-width (e. g., INT8) quantization can be employed to accelerate the inference process.
Large-Scale Training System for 100-Million Classification at Alibaba
In the last decades, extreme classification has become an essential topic for deep learning.
