Search Results for author:
Found 68 papers, 45 papers with code
Bridging Sensor Gaps via Single-Direction Tuning for Hyperspectral Image Classification
In this paper, aiming to solve this problem, we propose the single-direction tuning (SDT) strategy, which serves as a bridge, allowing us to leverage existing labeled HSI datasets even RGB datasets to enhance the performance on new HSI datasets with limited samples.
Unsupervised Open-Vocabulary Object Localization in Videos
In this paper, we show that recent advances in video representation learning and pre-trained vision-language models allow for substantial improvements in self-supervised video object localization.
Spiking-LEAF: A Learnable Auditory front-end for Spiking Neural Networks
Brain-inspired spiking neural networks (SNNs) have demonstrated great potential for temporal signal processing.
SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channels
Recently, many parameter-efficient fine-tuning (PEFT) methods have been proposed, and their experiments demonstrate that tuning only 1% of extra parameters could surpass full fine-tuning in low-data resource scenarios.
Dataset Condensation via Generative Model
Dataset condensation aims to condense a large dataset with a lot of training samples into a small set.
ViT-Lens: Towards Omni-modal Representations
A well-trained lens with a ViT backbone has the potential to serve as one of these foundation models, supervising the learning of subsequent modalities.
Ranked #1 on
Zero-shot 3D classification
on Objaverse LVIS
(using extra training data)
Recap: Detecting Deepfake Video with Unpredictable Tampered Traces via Recovering Faces and Mapping Recovered Faces
In the recovering stage, the model focuses on randomly masking regions of interest (ROIs) and reconstructing real faces without unpredictable tampered traces, resulting in a relatively good recovery effect for real faces while a poor recovery effect for fake faces.
Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks
Despite the rapid advancement of unsupervised learning in visual representation, it requires training on large-scale datasets that demand costly data collection, and pose additional challenges due to concerns regarding data privacy.
Revisiting Vision Transformer from the View of Path Ensemble
Therefore, we propose the path pruning and EnsembleScale skills for improvement, which cut out the underperforming paths and re-weight the ensemble components, respectively, to optimize the path combination and make the short paths focus on providing high-quality representation for subsequent paths.
DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models
To showcase the power of the proposed approach, we generate datasets with rich dense pixel-wise labels for a wide range of downstream tasks, including semantic segmentation, instance segmentation, and depth estimation.
UniVTG: Towards Unified Video-Language Temporal Grounding
Most methods in this direction develop taskspecific models that are trained with type-specific labels, such as moment retrieval (time interval) and highlight detection (worthiness curve), which limits their abilities to generalize to various VTG tasks and labels.
Ranked #1 on
Highlight Detection
on QVHighlights
(using extra training data)
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion
As such paired data is time-consuming and labor-intensive to acquire and restricted to a closed set, this potentially becomes the bottleneck for applications in an open world.
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone
Video-language pre-training (VLP) has become increasingly important due to its ability to generalize to various vision and language tasks.
GroundNLQ @ Ego4D Natural Language Queries Challenge 2023
Motivated by this, we leverage a two-stage pre-training strategy to train egocentric feature extractors and the grounding model on video narrations, and further fine-tune the model on annotated data.
TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter
In situations involving system upgrades that require updating the upstream foundation model, it becomes essential to re-train all downstream modules to adapt to the new foundation model, which is inflexible and inefficient.
AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn
2) Flexible inputs and intermediate results.
Too Large; Data Reduction for Vision-Language Pre-Training
Specifically, TL;DR can compress the mainstream VLP datasets at a high ratio, e. g., reduce well-cleaned CC3M dataset from 2. 82M to 0. 67M ($\sim$24\%) and noisy YFCC15M from 15M to 2. 5M ($\sim$16. 7\%).
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community.
VisorGPT: Learning Visual Prior via Generative Pre-Training
Experimental results demonstrate that VisorGPT can effectively model the visual prior, which can be employed for many vision tasks, such as customizing accurate human pose for conditional image synthesis models like ControlNet.
A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension
Most existing cross-modal language-to-video retrieval (VR) research focuses on single-modal input from video, i. e., visual representation, while the text is omnipresent in human environments and frequently critical to understand video.
HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video
Our method enables pausing the video at any frame and rendering all scene details (dynamic humans, objects, and backgrounds) from arbitrary viewpoints.
Open-World Weakly-Supervised Object Localization
To handle such data, we propose a novel paradigm of contrastive representation co-learning using both labeled and unlabeled data to generate a complete G-CAM (Generalized Class Activation Map) for object localization, without the requirement of bounding box annotation.
Representation Learning
Weakly-Supervised Object Localization
ICDAR 2023 Video Text Reading Competition for Dense and Small Text
In this competition report, we establish a video text reading benchmark, DSText, which focuses on dense and small text reading challenges in the video with various scenarios.
Attack is Good Augmentation: Towards Skeleton-Contrastive Representation Learning
In particular, we propose a novel Attack-Augmentation Mixing-Contrastive learning (A$^2$MC) to contrast hard positive features and hard negative features for learning more robust skeleton representations.
SparseFormer: Sparse Visual Recognition via Limited Latent Tokens
In this paper, we challenge this dense paradigm and present a new method, coined SparseFormer, to imitate human's sparse visual recognition in an end-to-end manner.
Sparse Representation-based Classification
Video Classification
Affordance Grounding from Demonstration Video to Target Image
Humans excel at learning from expert demonstrations and solving their own problems.
Ranked #1 on
Video-to-image Affordance Grounding
on EPIC-Hotspot
DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
In contrast, synthetic data can be freely available using a generative model (e. g., DALL-E, Stable Diffusion).
Making Vision Transformers Efficient from A Token Sparsification View
In this work, we propose a novel Semantic Token ViT (STViT), for efficient global and local vision transformers, which can also be revised to serve as backbone for downstream tasks.
Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm
In view of the practicality of PETL, previous works focus on tuning a small set of parameters for each downstream task in an end-to-end manner while rarely considering the task distribution shift issue between the pre-training task and the downstream task.
Mover: Mask and Recovery based Facial Part Consistency Aware Method for Deepfake Video Detection
Specifically, given a real face image, we first pretrain a masked autoencoder to learn facial part consistency by dividing faces into three parts and randomly masking ROIs, which are then recovered based on the unmasked facial parts.
Object-centric Learning with Cyclic Walks between Parts and Whole
To capture compositional entities of the scene, we proposed cyclic walks between perceptual features extracted from CNN or transformers and object entities.
STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognition
For the first time, we introduce vision Transformers into PPAR by treating a video as a tubelet sequence, and accordingly design two complementary mechanisms, i. e., sparsification and anonymization, to remove privacy from a spatio-temporal perspective.
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator.
MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering
To build Video Question Answering (VideoQA) systems capable of assisting humans in daily activities, seeking answers from long-form videos with diverse and complex events is a must.
Ranked #3 on
Video Question Answering
on STAR: Situated Reasoning
Position-guided Text Prompt for Vision-Language Pre-training
In this work, we propose a novel Position-guided Text Prompt (PTP) paradigm to enhance the visual grounding ability of cross-modal models trained with VLP.
Ranked #2 on
Zero-Shot Cross-Modal Retrieval
on COCO 2014
PV3D: A 3D Generative Model for Portrait Video Generation
While some prior works have applied such image GANs to unconditional 2D portrait video generation and static 3D portrait synthesis, there are few works successfully extending GANs for generating 3D-aware portrait videos.
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis
Vector-Quantized (VQ-based) generative models usually consist of two basic components, i. e., VQ tokenizers and generative transformers.
Learning to Learn: How to Continuously Teach Humans and Machines
For example, when we learn mathematics at school, we build upon our knowledge of addition to learn multiplication.
Darwinian Model Upgrades: Model Evolving with Selective Compatibility
The traditional model upgrading paradigm for retrieval requires recomputing all gallery embeddings before deploying the new model (dubbed as "backfilling"), which is quite expensive and time-consuming considering billions of instances in industrial applications.
Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task
However, CL on VQA involves not only the expansion of label sets (new Answer sets).
Single-Stage Open-world Instance Segmentation with Cross-task Consistency Regularization
Based on the single-stage instance segmentation framework, we propose a regularization model to predict foreground pixels and use its relation to instance segmentation to construct a cross-task consistency loss.
Egocentric Video-Language Pretraining @ Ego4D Challenge 2022
In this report, we propose a video-language pretraining (VLP) based solution \cite{kevin2022egovlp} for four Ego4D challenge tasks, including Natural Language Query (NLQ), Moment Query (MQ), Object State Change Classification (OSCC), and PNR Localization (PNR).
Egocentric Video-Language Pretraining @ EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022
In this report, we propose a video-language pretraining (VLP) based solution \cite{kevin2022egovlp} for the EPIC-KITCHENS-100 Multi-Instance Retrieval (MIR) challenge.
The Metaverse Data Deluge: What Can We Do About It?
In the Metaverse, the physical space and the virtual space co-exist, and interact simultaneously.
Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval
We surprisingly find that discrete text tokens coupled with a pretrained contrastive text model yields the best performance, which can even outperform state-of-the-art on the iVQA and How2QA datasets without additional training on millions of video-text data.
Ranked #1 on
Video Question Answering
on How2QA
Egocentric Video-Language Pretraining
Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention.
Ranked #2 on
Object State Change Classification
on Ego4D
Label-Efficient Online Continual Object Detection in Streaming Video
Remarkably, with only 25% annotated video frames, our method still outperforms the base CL learners, which are trained with 100% annotations on all video frames.
DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes
In this paper, we present DeVRF, a novel representation to accelerate learning dynamic radiance fields.
Novel View Synthesis for High-fidelity Headshot Scenes
Based on these observations, we propose a method to use both NeRF and 3DMM to synthesize a high-fidelity novel view of a scene with a face.
GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval
In this paper, we introduce a new dataset called Kinetic-GEB+.
Ranked #1 on
Text to Video Retrieval
on Kinetics-GEB+
(text-to-video R@1 metric)
Unified Transformer Tracker for Object Tracking
Although UniTrack \cite{wang2021different} demonstrates that a shared appearance model with multiple heads can be used to tackle individual tracking tasks, it fails to exploit the large-scale tracking datasets for training and performs poorly on single object tracking.
Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval
Recent dominant methods for video-language pre-training (VLP) learn transferable representations from the raw pixels in an end-to-end manner to achieve advanced performance on downstream video-language retrieval.
All in One: Exploring Unified Video-Language Pre-training
In this work, we for the first time introduce an end-to-end video-language model, namely \textit{all-in-one Transformer}, that embeds raw video and textual signals into joint representations using a unified backbone architecture.
Ranked #5 on
Video Question Answering
on STAR: Situated Reasoning
AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant
In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos to provide step-by-step help in the user's view.
Contrastive Learning of Semantic and Visual Representations for Text Tracking
Semantic representation is of great benefit to the video text tracking(VTT) task that requires simultaneously classifying, detecting, and tracking texts in the video.
Video-Text Pre-training with Learned Regions
Video-Text pre-training aims at learning transferable representations from large-scale video-text pairs via aligning the semantics between visual and textual information.
Object-aware Video-language Pre-training for Retrieval
In this work, we present Object-aware Transformers, an object-centric approach that extends video-language transformer to incorporate object representations.
AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant
In contrast, we present a new task called Task-oriented Question-driven Video Segment Retrieval (TQVSR).
AVA-AVD: Audio-Visual Speaker Diarization in the Wild
Audio-visual speaker diarization aims at detecting "who spoke when" using both auditory and visual signals.
MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learning
With such multi-dimension and multi-scale factorization, our MorphMLP block can achieve a great accuracy-computation balance.
Ranked #28 on
Action Recognition
on Something-Something V2
(using extra training data)
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
On Pursuit of Designing Multi-modal Transformer for Video Grounding
Almost all existing video grounding methods fall into two frameworks: 1) Top-down model: It predefines a set of segment candidates and then conducts segment classification and regression.
Deep Motion Prior for Weakly-Supervised Temporal Action Localization
In this paper, we analyze that the motion cues behind the optical flow features are complementary informative.
Optical Flow Estimation
Weakly-supervised Temporal Action Localization
+1
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers.
NUS-HLT Report for ActivityNet Challenge 2021 AVA (Speaker)
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers.
Generic Event Boundary Detection: A Benchmark for Event Segmentation
This paper presents a novel task together with a new benchmark for detecting generic, taxonomy-free event boundaries that segment a whole video into chunks.
Channel Augmented Joint Learning for Visible-Infrared Recognition
This paper introduces a powerful channel augmented joint learning strategy for the visible-infrared recognition problem.
Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization
We propose to explicitly model the Actor-Context-Actor Relation, which is the relation between two actors based on their interactions with the context.
Ranked #2 on
Action Recognition
on AVA v2.1
