Search Results for author:
Found 47 papers, 33 papers with code
DiT-Air: Revisiting the Efficiency of Diffusion Model Architecture Design in Text to Image Generation
In this work, we empirically study Diffusion Transformers (DiTs) for text-to-image generation, focusing on architectural choices, text-conditioning strategies, and training protocols.
DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning
Multimodal learning integrates complementary information from diverse modalities to enhance the decision-making process.
OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?
OVO-Bench evaluates the ability of video LLMs to reason and respond to events occurring at specific timestamps under three distinct scenarios: (1) Backward tracing: trace back to past events to answer the question.
Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction
Active Real-time interaction with video LLMs introduces a new paradigm for human-computer interaction, where the model not only understands user intent but also responds while continuously processing streaming video on the fly.
Reasoning to Attend: Try to Understand How <SEG> Token Works
In this work, we first visualize the similarity maps, which are obtained by computing the semantic similarity between the $\texttt{<SEG>}$ token and the image token embeddings derived from the last hidden layer in both the LLaVA encoder and SAM decoder.
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
Recent advancements in multimodal large language models (MLLMs) have made significant strides in open-world understanding.
SimC3D: A Simple Contrastive 3D Pretraining Framework Using RGB Images
To this end, we propose SimC3D, a simple but effective 3D contrastive learning framework, for the first time, pretraining 3D backbones from pure RGB image data.
SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree
Benefiting from its heuristic search design, SAM2Long is robust toward occlusions and object reappearances, and can effectively segment and track objects for complex long-term videos.
Ranked #4 on
Visual Object Tracking
on DiDi
Rethinking Image-to-Video Adaptation: An Object-centric Perspective
Instead of finetuning the entire image backbone, many image-to-video adaptation paradigms use lightweight adapters for temporal modeling on top of the spatial module.
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
This long-context capability allows IXC-2. 5 to excel in tasks requiring extensive input and output contexts.
Ranked #4 on
Video Question Answering
on TVBench
Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models
Building on this core model, we propose a greedy algorithm that grows the architecture into high-resolution end-to-end models, while preserving the integrity of the pre-trained representation, stabilizing training, and reducing the need for large high-resolution datasets.
Streaming Long Video Understanding with Large Language Models
After the encoding process, the Adaptive Memory Selection strategy selects a constant number of question-related memories from all the historical memories and feeds them into the LLM to generate informative responses.
SongComposer: A Large Language Model for Lyric and Melody Composition in Song Generation
We present SongComposer, an innovative LLM designed for song composition.
VideoPrism: A Foundational Visual Encoder for Video Understanding
We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model.
Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation
In this paper, we propose a simple yet effective approach for self-supervised video object segmentation (VOS).
Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos
In the second stage, for each semantics, we randomly sample slots from the corresponding Gaussian distribution and perform masked feature aggregation within the semantic area to exploit temporal correspondence patterns for instance identification.
Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation
Based on the STA score, we are able to progressively prune the tokens without introducing any additional parameters or requiring further re-training.
Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation
In this work, we propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture), to effectively capture the cross-modal audio-to-gesture associations and preserve temporal coherence for high-fidelity audio-driven co-speech gesture generation.
Motion-inductive Self-supervised Object Discovery in Videos
In this paper, we consider the task of unsupervised object discovery in videos.
Ranked #3 on
Unsupervised Object Segmentation
on DAVIS 2016
Static and Dynamic Concepts for Self-supervised Video Representation Learning
In this paper, we propose a novel learning scheme for self-supervised video representation learning.
Exploring Fine-Grained Audiovisual Categorization with the SSW60 Dataset
We thoroughly benchmark audiovisual classification performance and modality fusion experiments through the use of state-of-the-art transformer methods.
Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models
Utilizing vision and language models (VLMs) pre-trained on large-scale image-text pairs is becoming a promising paradigm for open-vocabulary visual recognition.
Ranked #1 on
Zero-Shot Action Recognition
on HMDB51
Dual Contrastive Learning for Spatio-temporal Representation
In this way, the static scene and the dynamic motion are simultaneously encoded into the compact RGB representation.
Controllable Augmentations for Video Representation Learning
The major reason is that the positive pairs, i. e., different clips sampled from the same video, have limited temporal receptive field, and usually share similar background but differ in motions.
Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation
To enhance the quality of synthesized gestures, we develop a contrastive learning strategy based on audio-text alignment for better audio representations.
Ranked #3 on
Gesture Generation
on TED Gesture Dataset
Visual Sound Localization in the Wild by Cross-Modal Interference Erasing
Specifically, we observe that the previous practice of learning only a single audio representation is insufficient due to the additive nature of audio signals.
Class-aware Sounding Objects Localization via Audiovisual Correspondence
To address this problem, we propose a two-stage step-by-step learning framework to localize and recognize sounding objects in complex audiovisual scenarios using only the correspondence between audio and vision.
Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision
Modern self-supervised learning algorithms typically enforce persistency of instance representations across views.
Exploring Temporal Granularity in Self-Supervised Video Representation Learning
The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips.
Motion-aware Contrastive Video Representation Learning via Foreground-background Merging
To alleviate such bias, we propose \textbf{F}oreground-b\textbf{a}ckground \textbf{Me}rging (FAME) to deliberately compose the moving foreground region of the selected video onto the static background of others.
Revisiting 3D ResNets for Video Recognition
A recent work from Bello shows that training and scaling strategies may be more significant than model architectures for visual recognition.
Ranked #40 on
Action Classification
on Kinetics-600
Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization
The crux of self-supervised video representation learning is to build general features from unlabeled videos.
TA2N: Two-Stage Action Alignment Network for Few-shot Action Recognition
The first stage locates the action by learning a temporal affine transform, which warps each video feature to its action duration while dismissing the action-irrelevant feature (e. g. background).
3D Object Detection for Autonomous Driving: A Survey
Autonomous driving is regarded as one of the most promising remedies to shield human beings from severe crashes.
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval.
Ranked #3 on
Zero-Shot Video Retrieval
on YouCook2
(text-to-video Mean Rank metric)
BADet: Boundary-Aware 3D Object Detection from Point Clouds
Specifically, instead of refining each proposal independently as previous works do, we represent each proposal as a node for graph construction within a given cut-off threshold, associating proposals in the form of local neighborhood graph, with boundary correlations of an object being explicitly exploited.
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation
Our baseline model outperforms the LVIS 2020 Challenge winning entry by +3. 6 mask AP on rare categories.
Ranked #1 on
Object Detection
on PASCAL VOC 2007
Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching
First, we propose to learn robust object representations by aggregating the candidate sound localization results in the single source scenes.
Finding Action Tubes with a Sparse-to-Dense Framework
The task of spatial-temporal action detection has attracted increasing attention among researchers.
Ranked #3 on
Action Detection
on UCF Sports
(Video-mAP 0.2 metric)
Spatiotemporal Contrastive Video Representation Learning
Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.
Ranked #1 on
Self-Supervised Action Recognition
on Kinetics-600
Multiple Sound Sources Localization from Coarse to Fine
How to visually localize multiple sound sources in unconstrained videos is a formidable problem, especially when lack of the pairwise sound-object annotations.
Human in Events: A Large-Scale Benchmark for Human-centric Video Analysis in Complex Events
To this end, we present a new large-scale dataset with comprehensive annotations, named Human-in-Events or HiEve (Human-centric video analysis in complex Events), for the understanding of human motions, poses, and actions in a variety of realistic events, especially in crowd & complex events.
End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection
Reliable and accurate 3D object detection is a necessity for safe autonomous driving.
ATRW: A Benchmark for Amur Tiger Re-identification in the Wild
This paper tries to fill the gap by introducing a novel large-scale dataset, the Amur Tiger Re-identification in the Wild (ATRW) dataset.
Weakly Supervised Scene Parsing with Point-based Distance Metric Learning
Semantic scene parsing is suffering from the fact that pixel-level annotations are hard to be collected.
Attentive Generative Adversarial Network for Raindrop Removal from a Single Image
This injection of visual attention to both generative and discriminative networks is the main contribution of this paper.
