Search Results for author:
Found 99 papers, 66 papers with code
RingID: Rethinking Tree-Ring Watermarking for Enhanced Multi-Key Identification
We revisit Tree-Ring Watermarking, a recent diffusion model watermarking method that demonstrates great robustness to various attacks.
Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion Models
This study explores the role of cross-attention during inference in text-conditional diffusion models.
Diffusion-Driven Self-Supervised Learning for Shape Reconstruction and Pose Estimation
Furthermore, we introduce a pretrain-to-refine self-supervised training paradigm to train our network.
DragAnything: Motion Control for Anything using Entity Representation
We introduce DragAnything, which utilizes a entity representation to achieve motion control for any object in controllable video generation.
Bring Your Own Character: A Holistic Solution for Automatic Facial Animation Generation of Customized Characters
In our solution, a deep learning model was first trained to retarget the facial expression from input face images to virtual human faces by estimating the blendshape coefficients.
Skip \n: A Simple Method to Reduce Hallucination in Large Vision-Language Models
Recent advancements in large vision-language models (LVLMs) have demonstrated impressive capability in visual information understanding with human language.
Delocate: Detection and Localization for Deepfake Videos with Randomly-Located Tampered Traces
Deepfake videos are becoming increasingly realistic, showing subtle tampering traces on facial areasthat vary between frames.
Towards A Better Metric for Text-to-Video Generation
Experiments on the TVGE dataset demonstrate the superiority of the proposed T2VScore on offering a better metric for text-to-video generation.
Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions
This work presents Moonshot, a new video generation model that conditions simultaneously on multimodal inputs of image and text.
COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training
\ModelName, our unified framework, merges unimodal and multimodal elements, enhancing model performance for tasks involving textual and visual data while notably reducing learnable parameters.
Parrot Captions Teach CLIP to Spot Text
Despite CLIP being the foundation model in numerous vision-language applications, the CLIP suffers from a severe text spotting bias.
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby boosting human productivity.
ShowRoom3D: Text to High-Quality 3D Room Generation Using 3D Priors
Previous methods using 2D diffusion priors to optimize neural radiance fields for generating room-scale scenes have shown unsatisfactory quality.
MAG-Edit: Localized Image Editing in Complex Scenarios via Mask-Based Attention-Adjusted Guidance
Recent diffusion-based image editing approaches have exhibited impressive editing capabilities in images with simple compositions.
Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator
Additionally, to validate the efficacy of generated data quantitatively, we add the instruction tuning data produced by Genixer into the training of two representative MLLMs and observe the consistent improvements on various VQA tasks and multimodal benchmarks.
X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model
To enhance the guidance ability of X-Adapter, we employ a null-text training strategy for the upgraded model.
ColonNeRF: High-Fidelity Neural Reconstruction of Long Colonoscopy
Specifically, to reconstruct the entire colon in a piecewise manner, our ColonNeRF introduces a region division and integration module, effectively reducing shape dissimilarity and ensuring geometric consistency in each segment.
VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence
In contrast to previous methods that rely on dense correspondences, we introduce the VideoSwap framework that exploits semantic point correspondences, inspired by our observation that only a small number of semantic points are necessary to align the subject's motion trajectory and modify its shape.
Bootstrapping SparseFormers from Vision Foundation Models
In this paper, we propose to bootstrap SparseFormers from ViT-based vision foundation models in a simple and efficient way.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
MLLMs-Augmented Visual-Language Representation Learning
Visual-language pre-training has achieved remarkable success in many multi-modal tasks, largely attributed to the availability of large-scale image-text datasets.
MD-Splatting: Learning Metric Deformation from 4D Gaussians in Highly Deformable Scenes
MD-Splatting builds on recent advances in Gaussian splatting, a method that learns the properties of a large number of Gaussians for state-of-the-art and fast novel view synthesis.
Continual Learning for Image Segmentation with Dynamic Query
Image segmentation based on continual learning exhibits a critical drop of performance, mainly due to catastrophic forgetting and background shift, as they are required to incorporate new classes continually.
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion.
ViT-Lens: Towards Omni-modal Representations
In this paper, we present ViT-Lens-2 that facilitates efficient omni-modal representation learning by perceiving novel modalities with a pretrained ViT and aligning them to a pre-defined space.
Paragraph-to-Image Generation with Information-Enriched Diffusion Model
In this paper, we introduce an information-enriched diffusion model for paragraph-to-image generation task, termed ParaDiffusion, which delves into the transference of the extensive semantic comprehension capabilities of large language models to the task of image generation.
Integrating View Conditions for Image Synthesis
In the field of image processing, applying intricate semantic modifications within existing images remains an enduring challenge.
CVPR 2023 Text Guided Video Editing Competition
In this paper we present a retrospective on the competition and describe the winning method.
DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing
To overcome this, we propose to introduce the dynamic Neural Radiance Fields (NeRF) as the innovative video representation, where the editing can be performed in the 3D spaces and propagated to the entire video via the deformation field.
MotionDirector: Motion Customization of Text-to-Video Diffusion Models
Given a set of video clips of the same motion concept, the task of Motion Customization is to adapt existing text-to-video diffusion models to generate videos with this motion.
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation
In this paper, we are the first to propose a hybrid model, dubbed as Show-1, which marries pixel-based and latent-based VDMs for text-to-video generation.
Ranked #2 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
Bridging Sensor Gaps via Single-Direction Tuning for Hyperspectral Image Classification
In this paper, aiming to solve this problem, we propose the single-direction tuning (SDT) strategy, which serves as a bridge, allowing us to leverage existing labeled HSI datasets even RGB datasets to enhance the performance on new HSI datasets with limited samples.
Spiking-LEAF: A Learnable Auditory front-end for Spiking Neural Networks
Brain-inspired spiking neural networks (SNNs) have demonstrated great potential for temporal signal processing.
Unsupervised Open-Vocabulary Object Localization in Videos
In this paper, we show that recent advances in video representation learning and pre-trained vision-language models allow for substantial improvements in self-supervised video object localization.
SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channels
Recently, many parameter-efficient fine-tuning (PEFT) methods have been proposed, and their experiments demonstrate that tuning only 1% of extra parameters could surpass full fine-tuning in low-data resource scenarios.
Dataset Condensation via Generative Model
Dataset condensation aims to condense a large dataset with a lot of training samples into a small set.
ViT-Lens: Initiating Omni-Modal Exploration through 3D Insights
A well-trained lens with a ViT backbone has the potential to serve as one of these foundation models, supervising the learning of subsequent modalities.
Ranked #2 on
Zero-Shot Transfer 3D Point Cloud Classification
on ModelNet40
(using extra training data)
Recap: Detecting Deepfake Video with Unpredictable Tampered Traces via Recovering Faces and Mapping Recovered Faces
In the recovering stage, the model focuses on randomly masking regions of interest (ROIs) and reconstructing real faces without unpredictable tampered traces, resulting in a relatively good recovery effect for real faces while a poor recovery effect for fake faces.
Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks
Despite the rapid advancement of unsupervised learning in visual representation, it requires training on large-scale datasets that demand costly data collection, and pose additional challenges due to concerns regarding data privacy.
Revisiting Vision Transformer from the View of Path Ensemble
Therefore, we propose the path pruning and EnsembleScale skills for improvement, which cut out the underperforming paths and re-weight the ensemble components, respectively, to optimize the path combination and make the short paths focus on providing high-quality representation for subsequent paths.
DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models
To showcase the power of the proposed approach, we generate datasets with rich dense pixel-wise labels for a wide range of downstream tasks, including semantic segmentation, instance segmentation, and depth estimation.
UniVTG: Towards Unified Video-Language Temporal Grounding
Most methods in this direction develop taskspecific models that are trained with type-specific labels, such as moment retrieval (time interval) and highlight detection (worthiness curve), which limits their abilities to generalize to various VTG tasks and labels.
Ranked #3 on
Natural Language Moment Retrieval
on TACoS
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion
As such paired data is time-consuming and labor-intensive to acquire and restricted to a closed set, this potentially becomes the bottleneck for applications in an open world.
Ranked #5 on
Conditional Text-to-Image Synthesis
on COCO-MIG
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone
Video-language pre-training (VLP) has become increasingly important due to its ability to generalize to various vision and language tasks.
GroundNLQ @ Ego4D Natural Language Queries Challenge 2023
Motivated by this, we leverage a two-stage pre-training strategy to train egocentric feature extractors and the grounding model on video narrations, and further fine-tune the model on annotated data.
TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter
In situations involving system upgrades that require updating the upstream foundation model, it becomes essential to re-train all downstream modules to adapt to the new foundation model, which is inflexible and inefficient.
AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn
2) Flexible inputs and intermediate results.
Too Large; Data Reduction for Vision-Language Pre-Training
Specifically, TL;DR can compress the mainstream VLP datasets at a high ratio, e. g., reduce well-cleaned CC3M dataset from 2. 82M to 0. 67M ($\sim$24\%) and noisy YFCC15M from 15M to 2. 5M ($\sim$16. 7\%).
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community.
VisorGPT: Learning Visual Prior via Generative Pre-Training
Experimental results demonstrate that VisorGPT can effectively model the visual prior, which can be employed for many vision tasks, such as customizing accurate human pose for conditional image synthesis models like ControlNet.
A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension
Most existing cross-modal language-to-video retrieval (VR) research focuses on single-modal input from video, i. e., visual representation, while the text is omnipresent in human environments and frequently critical to understand video.
HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video
Our method enables pausing the video at any frame and rendering all scene details (dynamic humans, objects, and backgrounds) from arbitrary viewpoints.
Open-World Weakly-Supervised Object Localization
To handle such data, we propose a novel paradigm of contrastive representation co-learning using both labeled and unlabeled data to generate a complete G-CAM (Generalized Class Activation Map) for object localization, without the requirement of bounding box annotation.
ICDAR 2023 Video Text Reading Competition for Dense and Small Text
In this competition report, we establish a video text reading benchmark, DSText, which focuses on dense and small text reading challenges in the video with various scenarios.
Attack is Good Augmentation: Towards Skeleton-Contrastive Representation Learning
In particular, we propose a novel Attack-Augmentation Mixing-Contrastive learning (A$^2$MC) to contrast hard positive features and hard negative features for learning more robust skeleton representations.
SparseFormer: Sparse Visual Recognition via Limited Latent Tokens
In this paper, we challenge this dense paradigm and present a new method, coined SparseFormer, to imitate human's sparse visual recognition in an end-to-end manner.
Sparse Representation-based Classification
Video Classification
Affordance Grounding from Demonstration Video to Target Image
Humans excel at learning from expert demonstrations and solving their own problems.
Ranked #1 on
Video-to-image Affordance Grounding
on EPIC-Hotspot
DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
In contrast, synthetic data can be freely available using a generative model (e. g., DALL-E, Stable Diffusion).
Making Vision Transformers Efficient from A Token Sparsification View
In this work, we propose a novel Semantic Token ViT (STViT), for efficient global and local vision transformers, which can also be revised to serve as backbone for downstream tasks.
Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm
In view of the practicality of PETL, previous works focus on tuning a small set of parameters for each downstream task in an end-to-end manner while rarely considering the task distribution shift issue between the pre-training task and the downstream task.
Mover: Mask and Recovery based Facial Part Consistency Aware Method for Deepfake Video Detection
Specifically, given a real face image, we first pretrain a masked autoencoder to learn facial part consistency by dividing faces into three parts and randomly masking ROIs, which are then recovered based on the unmasked facial parts.
Object-centric Learning with Cyclic Walks between Parts and Whole
To capture compositional entities of the scene, we proposed cyclic walks between perceptual features extracted from vision transformers and object entities.
STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognition
For the first time, we introduce vision Transformers into PPAR by treating a video as a tubelet sequence, and accordingly design two complementary mechanisms, i. e., sparsification and anonymization, to remove privacy from a spatio-temporal perspective.
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator.
MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering
To build Video Question Answering (VideoQA) systems capable of assisting humans in daily activities, seeking answers from long-form videos with diverse and complex events is a must.
Ranked #2 on
Video Question Answering
on AGQA 2.0 balanced
Position-guided Text Prompt for Vision-Language Pre-training
In this work, we propose a novel Position-guided Text Prompt (PTP) paradigm to enhance the visual grounding ability of cross-modal models trained with VLP.
Ranked #5 on
Zero-Shot Cross-Modal Retrieval
on COCO 2014
PV3D: A 3D Generative Model for Portrait Video Generation
While some prior works have applied such image GANs to unconditional 2D portrait video generation and static 3D portrait synthesis, there are few works successfully extending GANs for generating 3D-aware portrait videos.
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis
Vector-Quantized (VQ-based) generative models usually consist of two basic components, i. e., VQ tokenizers and generative transformers.
Learning to Learn: How to Continuously Teach Humans and Machines
For example, when we learn mathematics at school, we build upon our knowledge of addition to learn multiplication.
Darwinian Model Upgrades: Model Evolving with Selective Compatibility
The traditional model upgrading paradigm for retrieval requires recomputing all gallery embeddings before deploying the new model (dubbed as "backfilling"), which is quite expensive and time-consuming considering billions of instances in industrial applications.
Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task
However, CL on VQA involves not only the expansion of label sets (new Answer sets).
Single-Stage Open-world Instance Segmentation with Cross-task Consistency Regularization
Based on the single-stage instance segmentation framework, we propose a regularization model to predict foreground pixels and use its relation to instance segmentation to construct a cross-task consistency loss.
Egocentric Video-Language Pretraining @ EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022
In this report, we propose a video-language pretraining (VLP) based solution \cite{kevin2022egovlp} for the EPIC-KITCHENS-100 Multi-Instance Retrieval (MIR) challenge.
Egocentric Video-Language Pretraining @ Ego4D Challenge 2022
In this report, we propose a video-language pretraining (VLP) based solution \cite{kevin2022egovlp} for four Ego4D challenge tasks, including Natural Language Query (NLQ), Moment Query (MQ), Object State Change Classification (OSCC), and PNR Localization (PNR).
The Metaverse Data Deluge: What Can We Do About It?
In the Metaverse, the physical space and the virtual space co-exist, and interact simultaneously.
Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval
We surprisingly find that discrete text tokens coupled with a pretrained contrastive text model yields the best performance, which can even outperform state-of-the-art on the iVQA and How2QA datasets without additional training on millions of video-text data.
Ranked #1 on
Video Question Answering
on iVQA
Egocentric Video-Language Pretraining
Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention.
Label-Efficient Online Continual Object Detection in Streaming Video
Remarkably, with only 25% annotated video frames, our method still outperforms the base CL learners, which are trained with 100% annotations on all video frames.
Novel View Synthesis for High-fidelity Headshot Scenes
Based on these observations, we propose a method to use both NeRF and 3DMM to synthesize a high-fidelity novel view of a scene with a face.
DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes
In this paper, we present DeVRF, a novel representation to accelerate learning dynamic radiance fields.
GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval
In this paper, we introduce a new dataset called Kinetic-GEB+.
Ranked #1 on
Boundary Captioning
on Kinetics-GEB+
Unified Transformer Tracker for Object Tracking
Although UniTrack \cite{wang2021different} demonstrates that a shared appearance model with multiple heads can be used to tackle individual tracking tasks, it fails to exploit the large-scale tracking datasets for training and performs poorly on single object tracking.
Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval
Recent dominant methods for video-language pre-training (VLP) learn transferable representations from the raw pixels in an end-to-end manner to achieve advanced performance on downstream video-language retrieval.
All in One: Exploring Unified Video-Language Pre-training
In this work, we for the first time introduce an end-to-end video-language model, namely \textit{all-in-one Transformer}, that embeds raw video and textual signals into joint representations using a unified backbone architecture.
Ranked #6 on
TGIF-Transition
on TGIF-QA
(using extra training data)
AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant
In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos to provide step-by-step help in the user's view.
Contrastive Learning of Semantic and Visual Representations for Text Tracking
Semantic representation is of great benefit to the video text tracking(VTT) task that requires simultaneously classifying, detecting, and tracking texts in the video.
Video-Text Pre-training with Learned Regions
Video-Text pre-training aims at learning transferable representations from large-scale video-text pairs via aligning the semantics between visual and textual information.
Object-aware Video-language Pre-training for Retrieval
In this work, we present Object-aware Transformers, an object-centric approach that extends video-language transformer to incorporate object representations.
Ranked #20 on
Zero-Shot Video Retrieval
on DiDeMo
AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant
In contrast, we present a new task called Task-oriented Question-driven Video Segment Retrieval (TQVSR).
AVA-AVD: Audio-Visual Speaker Diarization in the Wild
Audio-visual speaker diarization aims at detecting "who spoke when" using both auditory and visual signals.
MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learning
With such multi-dimension and multi-scale factorization, our MorphMLP block can achieve a great accuracy-computation balance.
Ranked #38 on
Action Recognition
on Something-Something V2
(using extra training data)
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
On Pursuit of Designing Multi-modal Transformer for Video Grounding
Almost all existing video grounding methods fall into two frameworks: 1) Top-down model: It predefines a set of segment candidates and then conducts segment classification and regression.
Deep Motion Prior for Weakly-Supervised Temporal Action Localization
In this paper, we analyze that the motion cues behind the optical flow features are complementary informative.
Optical Flow Estimation
Weakly-supervised Temporal Action Localization
+1
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers.
NUS-HLT Report for ActivityNet Challenge 2021 AVA (Speaker)
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers.
Generic Event Boundary Detection: A Benchmark for Event Segmentation
This paper presents a novel task together with a new benchmark for detecting generic, taxonomy-free event boundaries that segment a whole video into chunks.
Channel Augmented Joint Learning for Visible-Infrared Recognition
This paper introduces a powerful channel augmented joint learning strategy for the visible-infrared recognition problem.
Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization
We propose to explicitly model the Actor-Context-Actor Relation, which is the relation between two actors based on their interactions with the context.
Ranked #2 on
Action Recognition
on AVA v2.1
