Search Results for author:
Found 21 papers, 11 papers with code
Adding Conditional Control to Text-to-Image Diffusion Models
ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls.
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
Once trained, the motion module can be inserted into a personalized T2I model to form a personalized animation generator.
SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models
The development of text-to-video (T2V), i. e., generating videos with a given text prompt, has been significantly advanced in recent years.
HotFlip: White-Box Adversarial Examples for Text Classification
We propose an efficient method to generate white-box adversarial examples to trick a character-level neural classifier.
A Local-to-Global Approach to Multi-modal Movie Scene Segmentation
Scene, as the crucial unit of storytelling in movies, contains complex activities of actors and their interactions in a physical environment.
AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation
To solve the above issues, we propose AutoGPart, a generic method enabling training generalizable 3D part segmentation networks with the task prior considered.
Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
Specifically, we construct a negative-sample-free triplet steam structure that is composed of an anchor stream without any masking, a spatial masking stream with Central Spatial Masking (CSM), and a temporal masking stream with Motion Attention Temporal Masking (MATM).
A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language
Although artificial intelligence (AI) has made significant progress in understanding molecules in a wide range of fields, existing models generally acquire the single cognitive ability from the single molecular modality.
Ranked #7 on
Molecule Captioning
on ChEBI-20
CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers
Although extensively studied for unimodal models, the acceleration for multimodal models, especially the vision-language Transformers, is relatively under-explored.
Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization
Specifically, 1) we maximize the MI between visual and semantic space for distribution alignment; 2) we leverage the temporal information for estimating the MI by encouraging MI to increase as more frames are observed.
Automatic Music Accompanist
The computer musician is able to produce musical accompaniment that relates musically to the human performance.
Sound Multimedia Audio and Speech Processing
MovieNet: A Holistic Dataset for Movie Understanding
We believe that such a holistic dataset would promote the researches on story-based long video understanding and beyond.
A Unified Framework for Shot Type Classification Based on Subject Centric Lens
Shots are key narrative elements of various videos, e. g. movies, TV series, and user-generated videos that are thriving over the Internet.
Online Multi-modal Person Search in Videos
The task of searching certain people in videos has seen increasing potential in real-world applications, such as video organization and editing.
BlockPlanner: City Block Generation With Vectorized Graph Representation
City modeling is the foundation for computational urban planning, navigation, and entertainment.
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering
The wide span of viewing positions within these scenes yields multi-scale renderings with very different levels of detail, which poses great challenges to neural radiance field and biases it towards compromised results.
Temporal and Contextual Transformer for Multi-Camera Editing of TV Shows
The ability to choose an appropriate camera view among multiple cameras plays a vital role in TV shows delivery.
Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production
Amateurs working on mini-films and short-form videos usually spend lots of time and effort on the multi-round complicated process of setting and adjusting scenes, plots, and cameras to deliver satisfying video shots.
HireVAE: An Online and Adaptive Factor Model Based on Hierarchical and Regime-Switch VAE
Factor model is a fundamental investment tool in quantitative investment, which can be empowered by deep learning to become more flexible and efficient in practical complicated investing situations.
Automated Conversion of Music Videos into Lyric Videos
Musicians and fans often produce lyric videos, a form of music videos that showcase the song's lyrics, for their favorite songs.
Cinematic Behavior Transfer via NeRF-based Differentiable Filming
In the evolving landscape of digital media and video production, the precise manipulation and reproduction of visual elements like camera movements and character actions are highly desired.
