Search Results for author:
Found 31 papers, 25 papers with code
InternVideo2: Scaling Video Foundation Models for Multimodal Video Understanding
We introduce InternVideo2, a new video foundation model (ViFM) that achieves the state-of-the-art performance in action recognition, video-text tasks, and video-centric dialogue.
Ranked #1 on
Audio Classification
on ESC-50
(using extra training data)
Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding
We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks.
Ranked #1 on
Temporal Action Localization
on FineAction
VideoMamba: State Space Model for Efficient Video Understanding
Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain.
Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition
Finally, we blend external multimodal knowledge in Adapt stage, by inserting multimodal knowledge adaptation modules into networks.
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents.
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM).
Vlogger: Make Your Dream A Vlog
More importantly, Vlogger can generate over 5-minute vlogs from open-world descriptions, without loss of video coherence on script and actor.
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
With the rapid development of Multi-modal Large Language Models (MLLMs), a number of diagnostic benchmarks have recently emerged to evaluate the comprehension capabilities of these models.
Ranked #1 on
Zero-Shot Video Question Answer
on STAR Benchmark
Harvest Video Foundation Models via Efficient Post-Pretraining
Building video-language foundation models is costly and difficult due to the redundant nature of video data and the lack of high-quality video-language datasets.
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Specifically, we utilize a multi-scale approach to generate video-related descriptions.
VideoChat: Chat-Centric Video Understanding
In this paper, we initiate an attempt of developing an end-to-end chat-centric video understanding system, coined as VideoChat.
Ranked #6 on
Video Question Answering
on MVBench
Video-based Generative Performance Benchmarking (Consistency)
Video-based Generative Performance Benchmarking (Contextual Understanding)
+5
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language
Different from existing interactive systems that rely on pure language, by incorporating pointing instructions, the proposed iGPT significantly improves the efficiency of communication between users and chatbots, as well as the accuracy of chatbots in vision-centric tasks, especially in complicated visual scenarios where the number of objects is greater than 2.
Unmasked Teacher: Towards Training-Efficient Video Foundation Models
Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain.
Ranked #1 on
Video Retrieval
on SSv2-template retrieval
(using extra training data)
UniFormerV2: Unlocking the Potential of Image ViTs for Video Understanding
The prolific performances of Vision Transformers (ViTs) in image tasks have prompted research into adapting the image ViTs for video tasks.
InternVideo: General Video Foundation Models via Generative and Discriminative Learning
Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications.
Ranked #1 on
Action Recognition
on Something-Something V1
(using extra training data)
UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer
UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format.
InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges
In this report, we present our champion solutions to five tracks at Ego4D challenge.
Ranked #1 on
State Change Object Detection
on Ego4D
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
On top of that, the performance of Tip-Adapter can be further boosted to be state-of-the-art on ImageNet by fine-tuning the cache model for 10$\times$ fewer epochs than existing methods, which is both effective and efficient.
MVP: Robust Multi-View Practice for Driving Action Localization
Due to the small data scale and unclear action boundary, the dataset presents a unique challenge to precisely localize all the different actions and classify their categories.
You Only Need 90K Parameters to Adapt Light: A Light Weight Transformer for Image Enhancement and Exposure Correction
Challenging illumination conditions (low-light, under-exposure and over-exposure) in the real world not only cast an unpleasant visual appearance but also taint the computer vision tasks.
Ranked #2 on
Image Enhancement
on Exposure-Errors
UniFormer: Unifying Convolution and Self-attention for Visual Recognition
Different from the typical transformer blocks, the relation aggregators in our UniFormer block are equipped with local and global token affinity respectively in shallow and deep layers, allowing to tackle both redundancy and dependency for efficient and effective representation learning.
Ranked #151 on
Image Classification
on ImageNet
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning
For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60. 9% and 71. 2% top-1 accuracy respectively.
PointCLIP: Point Cloud Understanding by CLIP
On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D.
Ranked #3 on
3D Open-Vocabulary Instance Segmentation
on STPLS3D
3D Open-Vocabulary Instance Segmentation
Few-Shot Learning
+6
Self-slimmed Vision Transformer
Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs by dynamic token aggregation.
MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learning
With such multi-dimension and multi-scale factorization, our MorphMLP block can achieve a great accuracy-computation balance.
Ranked #38 on
Action Recognition
on Something-Something V2
(using extra training data)
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification.
Pose-guided Generative Adversarial Net for Novel View Action Synthesis
We focus on the problem of novel-view human action synthesis.
Self-Slimming Vision Transformer
Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs by dynamic token aggregation.
UniFormer: Unified Transformer for Efficient Spatial-Temporal Representation Learning
For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60. 8% and 71. 4% top-1 accuracy respectively.
Ranked #8 on
Action Recognition
on Something-Something V1
CT-Net: Channel Tensorization Network for Video Classification
It can learn to exploit spatial, temporal and channel attention in a high-dimensional manner, to improve the cooperative power of all the feature dimensions in our CT-Module.
Ranked #18 on
Action Recognition
on Something-Something V1
End-to-End Object Detection with Adaptive Clustering Transformer
In this paper, a novel variant of transformer named Adaptive Clustering Transformer(ACT) has been proposed to reduce the computation cost for high-resolution input.
