Search Results for author:
Found 42 papers, 34 papers with code
Seed1.5-VL Technical Report
We present Seed1. 5-VL, a vision-language foundation model designed to advance general-purpose multimodal understanding and reasoning.
Make Your Training Flexible: Towards Deployment-Efficient Video Models
By making the sampling grid flexible and leveraging token selection, it is easily adopted in most popular video training frameworks, boosting model robustness with nearly no additional cost.
Ranked #5 on
Zero-Shot Video Retrieval
on MSR-VTT
(using extra training data)
TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
In the fostering stage (1-stage), we apply different LoRAs to fine-tune the diffusion model at different timestep intervals.
VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
This paper introduces a Hierarchical visual token Compression (HiCo) method designed for high-fidelity representation and a practical context modeling system VideoChat-Flash tailored for multimodal long-sequence processing.
Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
Current multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals though they give comprehensive perception and reasoning in a spectrum of vision applications.
Causal Diffusion Transformers for Generative Modeling
We introduce Causal Diffusion as the autoregressive (AR) counterpart of Diffusion models.
Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel
In this paper, we introduce a Self-Refining Data Flywheel (SRDF) that generates high-quality and large-scale navigational instruction-trajectory pairs by iteratively refining the data pool through the collaboration between two models, the instruction generator and the navigator, without any human-in-the-loop annotation.
TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
This paper proposes TimeSuite, a collection of new designs to adapt the existing short-form video MLLMs for long video understanding, including a simple yet efficient framework to process long video sequence, a high-quality video dataset for grounded tuning of MLLMs, and a carefully-designed instruction tuning task to explicitly incorporate the grounding supervision in the traditional QA format.
Ranked #7 on
Moment Retrieval
on Charades-STA
(using extra training data)
TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration
Vision-language foundation models (such as CLIP) have recently shown their power in transfer learning, owing to large-scale image-text pre-training.
MUSES: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration
Despite recent advancements in text-to-image generation, most existing methods struggle to create images with multiple objects and complex spatial relationships in the 3D world.
VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Model
With the growth of high-quality data and advancement in visual pre-training paradigms, Video Foundation Models (VFMs) have made significant progress recently, demonstrating their remarkable performance on traditional video understanding benchmarks.
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue.
Ranked #1 on
Action Classification
on MIT
Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding
We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks.
Ranked #2 on
Temporal Action Localization
on FineAction
VideoMamba: State Space Model for Efficient Video Understanding
Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain.
Ranked #58 on
Action Classification
on Kinetics-400
Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition
Finally, we blend external multimodal knowledge in Adapt stage, by inserting multimodal knowledge adaptation modules into networks.
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents.
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM).
Vlogger: Make Your Dream A Vlog
More importantly, Vlogger can generate over 5-minute vlogs from open-world descriptions, without loss of video coherence on script and actor.
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
With the rapid development of Multi-modal Large Language Models (MLLMs), a number of diagnostic benchmarks have recently emerged to evaluate the comprehension capabilities of these models.
Harvest Video Foundation Models via Efficient Post-Pretraining
Building video-language foundation models is costly and difficult due to the redundant nature of video data and the lack of high-quality video-language datasets.
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Specifically, we utilize a multi-scale approach to generate video-related descriptions.
VideoChat: Chat-Centric Video Understanding
In this paper, we initiate an attempt of developing an end-to-end chat-centric video understanding system, coined as VideoChat.
Ranked #3 on
Question Answering
on NExT-QA (Open-ended VideoQA)
Video-based Generative Performance Benchmarking (Consistency)
Video-based Generative Performance Benchmarking (Contextual Understanding)
+5
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language
Different from existing interactive systems that rely on pure language, by incorporating pointing instructions, the proposed iGPT significantly improves the efficiency of communication between users and chatbots, as well as the accuracy of chatbots in vision-centric tasks, especially in complicated visual scenarios where the number of objects is greater than 2.
Unmasked Teacher: Towards Training-Efficient Video Foundation Models
Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain.
Ranked #1 on
Video Retrieval
on SSv2-template retrieval
(using extra training data)
UniFormerV2: Unlocking the Potential of Image ViTs for Video Understanding
The prolific performances of Vision Transformers (ViTs) in image tasks have prompted research into adapting the image ViTs for video tasks.
InternVideo: General Video Foundation Models via Generative and Discriminative Learning
Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications.
Ranked #1 on
Action Recognition
on Something-Something V1
(using extra training data)
UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer
UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format.
InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges
In this report, we present our champion solutions to five tracks at Ego4D challenge.
Ranked #1 on
State Change Object Detection
on Ego4D
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
On top of that, the performance of Tip-Adapter can be further boosted to be state-of-the-art on ImageNet by fine-tuning the cache model for 10$\times$ fewer epochs than existing methods, which is both effective and efficient.
MVP: Robust Multi-View Practice for Driving Action Localization
Due to the small data scale and unclear action boundary, the dataset presents a unique challenge to precisely localize all the different actions and classify their categories.
You Only Need 90K Parameters to Adapt Light: A Light Weight Transformer for Image Enhancement and Exposure Correction
Challenging illumination conditions (low-light, under-exposure and over-exposure) in the real world not only cast an unpleasant visual appearance but also taint the computer vision tasks.
Ranked #3 on
Image Enhancement
on Exposure-Errors
UniFormer: Unifying Convolution and Self-attention for Visual Recognition
Different from the typical transformer blocks, the relation aggregators in our UniFormer block are equipped with local and global token affinity respectively in shallow and deep layers, allowing to tackle both redundancy and dependency for efficient and effective representation learning.
Ranked #154 on
Image Classification
on ImageNet
(Top 1 Accuracy metric)
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning
For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60. 9% and 71. 2% top-1 accuracy respectively.
PointCLIP: Point Cloud Understanding by CLIP
On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D.
Ranked #3 on
3D Open-Vocabulary Instance Segmentation
on STPLS3D
3D Open-Vocabulary Instance Segmentation
Few-Shot Learning
+7
Self-slimmed Vision Transformer
Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs by dynamic token aggregation.
MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learning
With such multi-dimension and multi-scale factorization, our MorphMLP block can achieve a great accuracy-computation balance.
Ranked #38 on
Action Recognition
on Something-Something V2
(using extra training data)
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification.
Pose-guided Generative Adversarial Net for Novel View Action Synthesis
We focus on the problem of novel-view human action synthesis.
Self-Slimming Vision Transformer
Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs by dynamic token aggregation.
UniFormer: Unified Transformer for Efficient Spatial-Temporal Representation Learning
For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60. 8% and 71. 4% top-1 accuracy respectively.
Ranked #9 on
Action Recognition
on Something-Something V1
CT-Net: Channel Tensorization Network for Video Classification
It can learn to exploit spatial, temporal and channel attention in a high-dimensional manner, to improve the cooperative power of all the feature dimensions in our CT-Module.
Ranked #19 on
Action Recognition
on Something-Something V1
End-to-End Object Detection with Adaptive Clustering Transformer
In this paper, a novel variant of transformer named Adaptive Clustering Transformer(ACT) has been proposed to reduce the computation cost for high-resolution input.
