Search Results for author:
Found 14 papers, 7 papers with code
OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation
To exploit the complementary nature of image and video data, we further propose a progressive training strategy, where OmniTokenizer is first trained on image data on a fixed resolution to develop the spatial encoding capacity and then jointly trained on image and video data on multiple resolutions to learn the temporal dynamics.
Ranked #10 on
Video Prediction
on Kinetics-600 12 frames, 64x64
OmniVid: A Generative Framework for Universal Video Understanding
The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution.
MouSi: Poly-Visual-Expert Vision-Language Models
This technique introduces a fusion network to unify the processing of outputs from different visual experts, while bridging the gap between image encoders and pre-trained LLMs.
Ranked #93 on
Visual Question Answering
on MM-Vet
To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning
Existing visual instruction tuning methods typically prompt large language models with textual descriptions to generate instruction-following data.
Ranked #82 on
Visual Question Answering
on MM-Vet
ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System
Existing deep video models are limited by specific tasks, fixed input-output spaces, and poor generalization capabilities, making it difficult to deploy them in real-world scenarios.
OmniTracker: Unifying Object Tracking by Tracking-with-Detection
Object tracking (OT) aims to estimate the positions of target objects in a video sequence.
Look Before You Match: Instance Understanding Matters in Video Object Segmentation
Towards this goal, we present a two-branch network for VOS, where the query-based instance segmentation (IS) branch delves into the instance details of the current frame and the VOS branch performs spatial-temporal matching with the memory bank.
Ranked #1 on
Semi-Supervised Video Object Segmentation
on Long Video Dataset
(using extra training data)
Fighting Malicious Media Data: A Survey on Tampering Detection and Deepfake Detection
Online media data, in the forms of images and videos, are becoming mainstream communication channels.
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture.
Ranked #4 on
Cross-Modal Retrieval
on Flickr30k
(using extra training data)
ObjectFormer for Image Manipulation Detection and Localization
Recent advances in image editing techniques have posed serious challenges to the trustworthiness of multimedia data, which drives the research of image tampering detection.
Efficient Video Transformers with Spatial-Temporal Token Selection
Video transformers have achieved impressive results on major video recognition benchmarks, which however suffer from high computational cost.
FT-TDR: Frequency-guided Transformer and Top-Down Refinement Network for Blind Face Inpainting
Blind face inpainting refers to the task of reconstructing visual contents without explicitly indicating the corrupted regions in a face image.
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection
The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images.
Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition
Humans can easily recognize actions with only a few examples given, while the existing video recognition models still heavily rely on the large-scale labeled data inputs.
Ranked #2 on
Few Shot Action Recognition
on Kinetics-100
