Search Results for author:
Found 153 papers, 95 papers with code
MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation
In this work, we address this by introducing MP-Mat, a novel 3D-and-instance-aware matting framework with multiplane representation, where the multiplane concept is designed from two different perspectives: scene geometry level and instance level.
AssistPDA: An Online Video Surveillance Assistant for Video Anomaly Prediction, Detection, and Analysis
To bridge this gap and facilitate the practical deployment of LLM-based VAD, we introduce AssistPDA, the first online video anomaly surveillance assistant that unifies video anomaly prediction, detection, and analysis (VAPDA) within a single framework.
Long-Context Autoregressive Video Modeling with Next-Frame Prediction
Long-context autoregressive modeling has significantly advanced language generation, but video generation still struggles to fully utilize extended temporal contexts.
Ranked #2 on
Video Generation
on UCF-101
Impossible Videos
In addition, a video benchmark is curated to assess Video-LLMs on their ability of understanding impossible videos, which particularly requires reasoning on temporal dynamics and world knowledge.
VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning
Videos, with their unique temporal dimension, demand precise grounded understanding, where answers are directly linked to visual, interpretable evidence.
Ranked #1 on
Zero-Shot Video Question Answer
on NExT-GQA
Edit Transfer: Learning Image Editing via Vision In-Context Relations
We introduce a new setting, Edit Transfer, where a model learns a transformation from just a single source-target example and applies it to a new query image.
TPDiff: Temporal Pyramid Video Diffusion Model
Given the inter-frame redundancy in video modality, maintaining full frame rates in high-entropy stages is unnecessary.
In-Context Defense in Computer Agents: An Empirical Study
Our findings highlight that (1) defensive reasoning must precede action planning for optimal performance, and (2) a minimal number of exemplars (fewer than three) is sufficient to induce an agent's defensive behavior.
Balanced Image Stylization with Style Matching Score
We present Style Matching Score (SMS), a novel optimization method for image stylization with diffusion models.
Automated Movie Generation via Multi-Agent CoT Planning
Existing long-form video generation frameworks lack automated planning, requiring manual input for storylines, scenes, cinematography, and character interactions, resulting in high costs and inefficiencies.
DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cycles
For tasks involving new paired knowledge, such as specific identities, a combination of a small set of paired image-text examples and larger-scale unpaired data is sufficient for effective domain-oriented adaptation.
Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models
At the core of our approach is Difix, a single-step image diffusion model trained to enhance and remove artifacts in rendered novel views caused by underconstrained regions of the 3D representation.
InterFeedback: Unveiling Interactive Intelligence of Large Multimodal Models via Human Feedback
Existing benchmarks do not test Large Multimodal Models (LMMs) on their interactive intelligence with human users, which is vital for developing general-purpose AI assistants.
PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data
Subsequently, we fine-tune this model with EditLoRA using a small, artist-curated dataset of before-and-after image pairs to capture distinct editing styles and techniques.
PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
These findings position PhysReason as a novel and comprehensive benchmark for evaluating physics-based reasoning capabilities in large language models.
WorldGUI: Dynamic Testing for Comprehensive Desktop GUI Automation
In addition, to address the challenges of dynamic GUI automation tasks, we propose GUI-Thinker, a holistic framework, leveraging a critique mechanism, that effectively manages the unpredictability and complexity of GUI interactions.
LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformer
Generating cognitive-aligned layered SVGs remains challenging due to existing methods' tendencies toward either oversimplified single-layer outputs or optimization-induced shape redundancies.
DiffSim: Taming Diffusion Models for Evaluating Visual Similarity
Diffusion models have fundamentally transformed the field of generative models, making the assessment of similarity between customized model outputs and reference inputs critically important.
IDProtector: An Adversarial Noise Encoder to Protect Against ID-Preserving Image Generation
Recently, zero-shot methods like InstantID have revolutionized identity-preserving generation.
VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting
They allow LLMs to guide the generation of visually-grounded text plans and textual-grounded video plans.
Anti-Reference: Universal and Immediate Defense Against Reference-Based Generation
Diffusion models have revolutionized generative modeling with their exceptional ability to produce high-fidelity images.
ROICtrl: Boosting Instance Control for Visual Generation
Natural language often struggles to accurately associate positional and attribute information with multiple instances, which limits current text-based visual generation models to simpler compositions featuring only a few dominant instances.
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
In this work, we develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations: (i) UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks; (ii) Interleaved Vision-Language-Action Streaming that flexibly unifies diverse needs within GUI tasks, enabling effective management of visual-action history in navigation or pairing multi-turn query-action sequences per screenshot to enhance training efficiency; (iii) Small-scale High-quality GUI Instruction-following Datasets by careful data curation and employing a resampling strategy to address significant data type imbalances.
Ranked #7 on
Natural Language Visual Grounding
on ScreenSpot
Factorized Visual Tokenization and Generation
Visual tokenizers are fundamental to image generation.
MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation
Recent advancements in video generation models, like Stable Video Diffusion, show promising results, but primarily focus on short, single-scene videos.
FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data
Fine-tuning MLLMs with Federated Learning (FL) allows for expanding the training data scope by including private data sources, thereby enhancing their practical applicability in privacy-sensitive domains.
The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use
The recently released model, Claude 3. 5 Computer Use, stands out as the first frontier AI model to offer computer use in public beta as a graphical user interface (GUI) agent.
ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning
In this paper, we present ReCapture, a method for generating new videos with novel camera trajectories from a single user-provided video.
Skinned Motion Retargeting with Dense Geometric Interaction Perception
Subsequently, we develop a novel spatio-temporal representation called the dense mesh interaction (DMI) field.
ControLRM: Fast and Controllable 3D Generation via Large Reconstruction Model
In the condition training branch, we lock the triplane decoder and reuses the deep and robust encoding layers pretrained with millions of 3D data in LRM.
EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models
Our experiments with extensive data indicate that the model trained on generated data of the advanced model can approximate its generation capability.
Image Watermarks are Removable Using Controllable Regeneration from Clean Noise
This scheme adds varying numbers of noise steps to the latent representation of the watermarked image, followed by a controlled denoising process starting from this noisy latent representation.
Unsupervised Prior Learning: Discovering Categorical Pose Priors from Videos
PPL uses a hierarchical memory to store compositional parts of prototypical poses, from which we distill a general pose prior.
One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos
We introduce VideoLISA, a video-based multimodal large language model designed to tackle the problem of language-instructed reasoning segmentation in videos.
High Quality Human Image Animation using Regional Supervision and Motion Blur Condition
Although generating reasonable results, existing methods often overlook the need for regional supervision in crucial areas such as the face and hands, and neglect the explicit modeling for motion blur, leading to unrealistic low-quality synthesis.
DOTA: Distributional Test-Time Adaptation of Vision-Language Models
Instead of naively memorizing representative test samples, Dota continually estimates the distributions of test samples, allowing the model to continually adapt to the deployment environment.
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation
Our method, VideoLLM-MoD, is inspired by mixture-of-depths LLMs and addresses the challenge of numerous vision tokens in long-term or streaming video.
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
We present a unified transformer, i. e., Show-o, that unifies multimodal understanding and generation.
Ranked #10 on
Text-to-Image Generation
on GenEval
Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach
As online video content rapidly grows, the task of text-video retrieval (TVR) becomes increasingly important.
Learning Video Context as Interleaved Multimodal Sequences
In this paper, we introduce MovieSeq, a multimodal language model developed to address the wide range of challenges in understanding video contexts.
Apprenticeship-Inspired Elegance: Synergistic Knowledge Distillation Empowers Spiking Neural Networks for Efficient Single-Eye Emotion Recognition
We introduce a novel multimodality synergistic knowledge distillation scheme tailored for efficient single-eye motion recognition tasks.
GUI Action Narrator: Where and When Did That Action Take Place?
The advent of Multimodal LLMs has significantly enhanced image OCR recognition capabilities, making GUI automation a viable reality for increasing efficiency in digital tasks.
VideoLLM-online: Online Video Large Language Model for Streaming Video
Recent Large Language Models have been enhanced with vision capabilities, enabling them to comprehend images, videos, and interleaved vision-language content.
VideoGUI: A Benchmark for GUI Automation from Instructional Videos
Graphical User Interface (GUI) automation holds significant promise for enhancing human productivity by assisting with computer tasks.
Steganalysis on Digital Watermarking: Is Your Defense Truly Impervious?
Digital watermarking techniques are crucial for copyright protection and source identification of images, especially in the era of generative AI models.
WMAdapter: Adding WaterMark Control to Latent Diffusion Models
Watermarking is crucial for protecting the copyright of AI-generated images.
ProcessPainter: Learn Painting Process from Sequence Data
The painting process of artists is inherently stepwise and varies significantly among different painters and styles.
Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning
Training models with longer in-context lengths is a significant challenge for multimodal model due to substantial GPU memory and computational costs.
Visual Perception by Large Language Model's Weights
Following this paradigm, we propose VLoRA with the perceptual weights generator.
Multi-Modal Generative Embedding Model
Existing models usually tackle these two types of problems by decoupling language modules into a text decoder for generation, and a text encoder for embedding.
LOVA3: Learning to Visual Question Answering, Asking and Assessment
Our results demonstrate consistent performance gains, underscoring the critical role of these additional tasks in fostering comprehensive intelligence in MLLMs.
Ranked #158 on
Visual Question Answering
on MM-Vet
Hallucination of Multimodal Large Language Models: A Survey
By drawing the granular classification and landscapes of hallucination causes, evaluation benchmarks, and mitigation methods, this survey aims to deepen the understanding of hallucinations in MLLMs and inspire further advancements in the field.
Learning Long-form Video Prior via Generative Pre-Training
Instead of operating on pixel space, it is efficient to employ visual locations like bounding boxes and keypoints to represent key information in videos, which can be simply discretized and then tokenized for consumption by GPT.
RingID: Rethinking Tree-Ring Watermarking for Enhanced Multi-Key Identification
We revisit Tree-Ring Watermarking, a recent diffusion model watermarking method that demonstrates great robustness to various attacks.
Faster Diffusion via Temporal Attention Decomposition
We explore the role of attention mechanism during inference in text-conditional diffusion models.
Diffusion-Driven Self-Supervised Learning for Shape Reconstruction and Pose Estimation
Furthermore, we introduce a pretrain-to-refine self-supervised training paradigm to train our network.
DragAnything: Motion Control for Anything using Entity Representation
We introduce DragAnything, which utilizes a entity representation to achieve motion control for any object in controllable video generation.
Bring Your Own Character: A Holistic Solution for Automatic Facial Animation Generation of Customized Characters
In our solution, a deep learning model was first trained to retarget the facial expression from input face images to virtual human faces by estimating the blendshape coefficients.
Skip \n: A Simple Method to Reduce Hallucination in Large Vision-Language Models
Recent advancements in large vision-language models (LVLMs) have demonstrated impressive capability in visual information understanding with human language.
Delocate: Detection and Localization for Deepfake Videos with Randomly-Located Tampered Traces
Deepfake videos are becoming increasingly realistic, showing few tampering traces on facial areasthat vary between frames.
Towards A Better Metric for Text-to-Video Generation
Experiments on the TVGE dataset demonstrate the superiority of the proposed T2VScore on offering a better metric for text-to-video generation.
Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions
This work presents Moonshot, a new video generation model that conditions simultaneously on multimodal inputs of image and text.
AssistGUI: Task-Oriented PC Graphical User Interface Automation
Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks thereby boosting human productivity.
Tune-An-Ellipse: CLIP Has Potential to Find What You Want
Visual prompting of large vision language models such as CLIP exhibits intriguing zero-shot capabilities.
L4D-Track: Language-to-4D Modeling Towards 6-DoF Tracking and Shape Reconstruction in 3D Point Cloud Stream
To this end we present a generic Language-to-4D modeling paradigm termed L4D-Track that tackles zero-shot 6-DoF \underline Track ing and shape reconstruction by learning pairwise implicit 3D representation and multi-level multi-modal alignment.
COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training
\ModelName, our unified framework, merges unimodal and multimodal elements, enhancing model performance for tasks involving textual and visual data while notably reducing learnable parameters.
Parrot Captions Teach CLIP to Spot Text
Despite CLIP being the foundation model in numerous vision-language applications, the CLIP suffers from a severe text spotting bias.
ShowRoom3D: Text to High-Quality 3D Room Generation Using 3D Priors
Previous methods using 2D diffusion priors to optimize neural radiance fields for generating room-scale scenes have shown unsatisfactory quality.
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby boosting human productivity.
MAG-Edit: Localized Image Editing in Complex Scenarios via Mask-Based Attention-Adjusted Guidance
Recent diffusion-based image editing approaches have exhibited impressive editing capabilities in images with simple compositions.
Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator
Multimodal Large Language Models (MLLMs) demonstrate exceptional problem-solving capabilities, but few research studies aim to gauge the ability to generate visual instruction tuning data.
Bootstrapping SparseFormers from Vision Foundation Models
In this paper, we propose to bootstrap SparseFormers from ViT-based vision foundation models in a simple and efficient way.
ColonNeRF: High-Fidelity Neural Reconstruction of Long Colonoscopy
Specifically, to reconstruct the entire colon in a piecewise manner, our ColonNeRF introduces a region division and integration module, effectively reducing shape dissimilarity and ensuring geometric consistency in each segment.
VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence
In contrast to previous methods that rely on dense correspondences, we introduce the VideoSwap framework that exploits semantic point correspondences, inspired by our observation that only a small number of semantic points are necessary to align the subject's motion trajectory and modify its shape.
X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model
To enhance the guidance ability of X-Adapter, we employ a null-text training strategy for the upgraded model.
DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation
Just as object pose estimation and tracking have aided robots for rigid manipulation, dense 3D tracking (scene flow) of highly deformable objects will enable new applications in robotics while aiding existing approaches, such as imitation learning or creating digital twins with real2sim transfer.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
MLLMs-Augmented Visual-Language Representation Learning
Visual-language pre-training has achieved remarkable success in many multi-modal tasks, largely attributed to the availability of large-scale image-text datasets.
Continual Learning for Image Segmentation with Dynamic Query
Image segmentation based on continual learning exhibits a critical drop of performance, mainly due to catastrophic forgetting and background shift, as they are required to incorporate new classes continually.
ViT-Lens: Towards Omni-modal Representations
In this paper, we present ViT-Lens-2 that facilitates efficient omni-modal representation learning by perceiving novel modalities with a pretrained ViT and aligning them to a pre-defined space.
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion.
Paragraph-to-Image Generation with Information-Enriched Diffusion Model
In this paper, we introduce an information-enriched diffusion model for paragraph-to-image generation task, termed ParaDiffusion, which delves into the transference of the extensive semantic comprehension capabilities of large language models to the task of image generation.
CVPR 2023 Text Guided Video Editing Competition
In this paper we present a retrospective on the competition and describe the winning method.
DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing
To overcome this, we propose to introduce the dynamic Neural Radiance Fields (NeRF) as the innovative video representation, where the editing can be performed in the 3D spaces and propagated to the entire video via the deformation field.
MotionDirector: Motion Customization of Text-to-Video Diffusion Models
Given a set of video clips of the same motion concept, the task of Motion Customization is to adapt existing text-to-video diffusion models to generate videos with this motion.
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation
In this paper, we are the first to propose a hybrid model, dubbed as Show-1, which marries pixel-based and latent-based VDMs for text-to-video generation.
Ranked #2 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
Unsupervised Open-Vocabulary Object Localization in Videos
In this paper, we show that recent advances in video representation learning and pre-trained vision-language models allow for substantial improvements in self-supervised video object localization.
Spiking-LEAF: A Learnable Auditory front-end for Spiking Neural Networks
Brain-inspired spiking neural networks (SNNs) have demonstrated great potential for temporal signal processing.
SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channels
Experiments on 19 visual transfer learning downstream tasks demonstrate that our SCT outperforms full fine-tuning on 18 out of 19 tasks by adding only 0. 11M parameters of the ViT-B, which is 780$\times$ fewer than its full fine-tuning counterpart.
Dataset Condensation via Generative Model
Dataset condensation aims to condense a large dataset with a lot of training samples into a small set.
ViT-Lens: Initiating Omni-Modal Exploration through 3D Insights
A well-trained lens with a ViT backbone has the potential to serve as one of these foundation models, supervising the learning of subsequent modalities.
Ranked #2 on
Zero-Shot Transfer 3D Point Cloud Classification
on ModelNet40
(using extra training data)
Recap: Detecting Deepfake Video with Unpredictable Tampered Traces via Recovering Faces and Mapping Recovered Faces
In the recovering stage, the model focuses on randomly masking regions of interest (ROIs) and reconstructing real faces without unpredictable tampered traces, resulting in a relatively good recovery effect for real faces while a poor recovery effect for fake faces.
Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks
Despite the rapid advancement of unsupervised learning in visual representation, it requires training on large-scale datasets that demand costly data collection, and pose additional challenges due to concerns regarding data privacy.
Revisiting Vision Transformer from the View of Path Ensemble
Therefore, we propose the path pruning and EnsembleScale skills for improvement, which cut out the underperforming paths and re-weight the ensemble components, respectively, to optimize the path combination and make the short paths focus on providing high-quality representation for subsequent paths.
DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models
To showcase the power of the proposed approach, we generate datasets with rich dense pixel-wise labels for a wide range of downstream tasks, including semantic segmentation, instance segmentation, and depth estimation.
UniVTG: Towards Unified Video-Language Temporal Grounding
Most methods in this direction develop taskspecific models that are trained with type-specific labels, such as moment retrieval (time interval) and highlight detection (worthiness curve), which limits their abilities to generalize to various VTG tasks and labels.
Ranked #8 on
Natural Language Moment Retrieval
on TACoS
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion
As such paired data is time-consuming and labor-intensive to acquire and restricted to a closed set, this potentially becomes the bottleneck for applications in an open world.
Ranked #5 on
Conditional Text-to-Image Synthesis
on COCO-MIG
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone
Video-language pre-training (VLP) has become increasingly important due to its ability to generalize to various vision and language tasks.
GroundNLQ @ Ego4D Natural Language Queries Challenge 2023
Motivated by this, we leverage a two-stage pre-training strategy to train egocentric feature extractors and the grounding model on video narrations, and further fine-tune the model on annotated data.
TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter
In situations involving system upgrades that require updating the upstream foundation model, it becomes essential to re-train all downstream modules to adapt to the new foundation model, which is inflexible and inefficient.
AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn
2) Flexible inputs and intermediate results.
Too Large; Data Reduction for Vision-Language Pre-Training
Specifically, TL;DR can compress the mainstream VLP datasets at a high ratio, e. g., reduce well-cleaned CC3M dataset from 2. 82M to 0. 67M ($\sim$24\%) and noisy YFCC15M from 15M to 2. 5M ($\sim$16. 7\%).
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community.
VisorGPT: Learning Visual Prior via Generative Pre-Training
Experimental results demonstrate that VisorGPT can effectively model the visual prior, which can be employed for many vision tasks, such as customizing accurate human pose for conditional image synthesis models like ControlNet.
A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension
Most existing cross-modal language-to-video retrieval (VR) research focuses on single-modal input from video, i. e., visual representation, while the text is omnipresent in human environments and frequently critical to understand video.
HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video
Our method enables pausing the video at any frame and rendering all scene details (dynamic humans, objects, and backgrounds) from arbitrary viewpoints.
Open-World Weakly-Supervised Object Localization
To handle such data, we propose a novel paradigm of contrastive representation co-learning using both labeled and unlabeled data to generate a complete G-CAM (Generalized Class Activation Map) for object localization, without the requirement of bounding box annotation.
ICDAR 2023 Video Text Reading Competition for Dense and Small Text
In this competition report, we establish a video text reading benchmark, DSText, which focuses on dense and small text reading challenges in the video with various scenarios.
SparseFormer: Sparse Visual Recognition via Limited Latent Tokens
In this paper, we challenge this dense paradigm and present a new method, coined SparseFormer, to imitate human's sparse visual recognition in an end-to-end manner.
Sparse Representation-based Classification
Video Classification
Affordance Grounding from Demonstration Video to Target Image
Humans excel at learning from expert demonstrations and solving their own problems.
Ranked #1 on
Video-to-image Affordance Grounding
on EPIC-Hotspot
DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
In contrast, synthetic data can be freely available using a generative model (e. g., DALL-E, Stable Diffusion).
Making Vision Transformers Efficient from A Token Sparsification View
In this work, we propose a novel Semantic Token ViT (STViT), for efficient global and local vision transformers, which can also be revised to serve as backbone for downstream tasks.
Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm
In view of the practicality of PETL, previous works focus on tuning a small set of parameters for each downstream task in an end-to-end manner while rarely considering the task distribution shift issue between the pre-training task and the downstream task.
Mover: Mask and Recovery based Facial Part Consistency Aware Method for Deepfake Video Detection
Specifically, given a real face image, we first pretrain a masked autoencoder to learn facial part consistency by dividing faces into three parts and randomly masking ROIs, which are then recovered based on the unmasked facial parts.
Object-centric Learning with Cyclic Walks between Parts and Whole
To capture compositional entities of the scene, we proposed cyclic walks between perceptual features extracted from vision transformers and object entities.
STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognition
For the first time, we introduce vision Transformers into PPAR by treating a video as a tubelet sequence, and accordingly design two complementary mechanisms, i. e., sparsification and anonymization, to remove privacy from a spatio-temporal perspective.
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator.
Position-guided Text Prompt for Vision-Language Pre-training
In this work, we propose a novel Position-guided Text Prompt (PTP) paradigm to enhance the visual grounding ability of cross-modal models trained with VLP.
Ranked #5 on
Zero-Shot Cross-Modal Retrieval
on COCO 2014
MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering
To build Video Question Answering (VideoQA) systems capable of assisting humans in daily activities, seeking answers from long-form videos with diverse and complex events is a must.
Ranked #2 on
Video Question Answering
on AGQA 2.0 balanced
PV3D: A 3D Generative Model for Portrait Video Generation
While some prior works have applied such image GANs to unconditional 2D portrait video generation and static 3D portrait synthesis, there are few works successfully extending GANs for generating 3D-aware portrait videos.
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis
Vector-Quantized (VQ-based) generative models usually consist of two basic components, i. e., VQ tokenizers and generative transformers.
Learning to Learn: How to Continuously Teach Humans and Machines
For example, when we learn mathematics at school, we build upon our knowledge of addition to learn multiplication.
Darwinian Model Upgrades: Model Evolving with Selective Compatibility
The traditional model upgrading paradigm for retrieval requires recomputing all gallery embeddings before deploying the new model (dubbed as "backfilling"), which is quite expensive and time-consuming considering billions of instances in industrial applications.
Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task
However, CL on VQA involves not only the expansion of label sets (new Answer sets).
Single-Stage Open-world Instance Segmentation with Cross-task Consistency Regularization
Based on the single-stage instance segmentation framework, we propose a regularization model to predict foreground pixels and use its relation to instance segmentation to construct a cross-task consistency loss.
Egocentric Video-Language Pretraining @ Ego4D Challenge 2022
In this report, we propose a video-language pretraining (VLP) based solution \cite{kevin2022egovlp} for four Ego4D challenge tasks, including Natural Language Query (NLQ), Moment Query (MQ), Object State Change Classification (OSCC), and PNR Localization (PNR).
Egocentric Video-Language Pretraining @ EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022
In this report, we propose a video-language pretraining (VLP) based solution \cite{kevin2022egovlp} for the EPIC-KITCHENS-100 Multi-Instance Retrieval (MIR) challenge.
The Metaverse Data Deluge: What Can We Do About It?
In the Metaverse, the physical space and the virtual space co-exist, and interact simultaneously.
Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval
We surprisingly find that discrete text tokens coupled with a pretrained contrastive text model yields the best performance, which can even outperform state-of-the-art on the iVQA and How2QA datasets without additional training on millions of video-text data.
Ranked #1 on
Video Question Answering
on iVQA
Egocentric Video-Language Pretraining
Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention.
Label-Efficient Online Continual Object Detection in Streaming Video
Remarkably, with only 25% annotated video frames, our method still outperforms the base CL learners, which are trained with 100% annotations on all video frames.
DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes
In this paper, we present DeVRF, a novel representation to accelerate learning dynamic radiance fields.
Novel View Synthesis for High-fidelity Headshot Scenes
Based on these observations, we propose a method to use both NeRF and 3DMM to synthesize a high-fidelity novel view of a scene with a face.
GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval
In this paper, we introduce a new dataset called Kinetic-GEB+.
Ranked #1 on
Boundary Captioning
on Kinetics-GEB+
Unified Transformer Tracker for Object Tracking
Although UniTrack \cite{wang2021different} demonstrates that a shared appearance model with multiple heads can be used to tackle individual tracking tasks, it fails to exploit the large-scale tracking datasets for training and performs poorly on single object tracking.
Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval
Recent dominant methods for video-language pre-training (VLP) learn transferable representations from the raw pixels in an end-to-end manner to achieve advanced performance on downstream video-language retrieval.
All in One: Exploring Unified Video-Language Pre-training
In this work, we for the first time introduce an end-to-end video-language model, namely \textit{all-in-one Transformer}, that embeds raw video and textual signals into joint representations using a unified backbone architecture.
Ranked #6 on
TGIF-Transition
on TGIF-QA
(using extra training data)
AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant
In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos to provide step-by-step help in the user's view.
Contrastive Learning of Semantic and Visual Representations for Text Tracking
Semantic representation is of great benefit to the video text tracking(VTT) task that requires simultaneously classifying, detecting, and tracking texts in the video.
Video-Text Pre-training with Learned Regions
Video-Text pre-training aims at learning transferable representations from large-scale video-text pairs via aligning the semantics between visual and textual information.
Object-aware Video-language Pre-training for Retrieval
In this work, we present Object-aware Transformers, an object-centric approach that extends video-language transformer to incorporate object representations.
Ranked #21 on
Zero-Shot Video Retrieval
on DiDeMo
AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant
In contrast, we present a new task called Task-oriented Question-driven Video Segment Retrieval (TQVSR).
AVA-AVD: Audio-Visual Speaker Diarization in the Wild
Audio-visual speaker diarization aims at detecting "who spoke when" using both auditory and visual signals.
MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learning
With such multi-dimension and multi-scale factorization, our MorphMLP block can achieve a great accuracy-computation balance.
Ranked #38 on
Action Recognition
on Something-Something V2
(using extra training data)
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Deep Motion Prior for Weakly-Supervised Temporal Action Localization
In this paper, we analyze that the motion cues behind the optical flow features are complementary informative.
Optical Flow Estimation
Weakly-supervised Temporal Action Localization
+1
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers.
Active Speaker Detection
Audio-Visual Active Speaker Detection
NUS-HLT Report for ActivityNet Challenge 2021 AVA (Speaker)
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers.
Active Speaker Detection
Audio-Visual Active Speaker Detection
Generic Event Boundary Detection: A Benchmark for Event Segmentation
This paper presents a novel task together with a new benchmark for detecting generic, taxonomy-free event boundaries that segment a whole video into chunks.
Channel Augmented Joint Learning for Visible-Infrared Recognition
This paper introduces a powerful channel augmented joint learning strategy for the visible-infrared recognition problem.
Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization
We propose to explicitly model the Actor-Context-Actor Relation, which is the relation between two actors based on their interactions with the context.
Ranked #2 on
Action Recognition
on AVA v2.1
