Search Results for author:
Found 30 papers, 24 papers with code
MiMo-VL Technical Report
We open-source MiMo-VL-7B-SFT and MiMo-VL-7B-RL, two powerful vision-language models delivering state-of-the-art performance in both general visual understanding and multimodal reasoning.
MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining
We present MiMo-7B, a large language model born for reasoning tasks, with optimization across both pre-training and post-training stages.
TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos
Remarkably, our experiments demonstrate that DTD achieves an 82. 8% reduction in video tokens while maintaining 98% performance on StreamingBench, revealing that over 80% of visual content in streaming videos is naturally redundant without requiring language guidance.
TEMPLE:Temporal Preference Learning of Video LLMs via Difficulty Scheduling and Pre-SFT Alignment
We further analyze the transferability of DPO data across architectures and the role of difficulty scheduling in optimization.
Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation
Autoregressive visual generation models typically rely on tokenizers to compress images into tokens that can be predicted sequentially.
UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?
To evaluate the performance of automatic metrics in unified AIGV evaluation, we introduce a benchmark called UVE-Bench.
Next Block Prediction: Video Generation via Semi-Auto-Regressive Modeling
Our model achieves FVD scores of 103. 3 on UCF101 and 25. 5 on K600, outperforming the vanilla NTP model by an average of 4. 4.
Parallelized Autoregressive Visual Generation
Autoregressive models have emerged as a powerful approach for visual generation but suffer from slow inference speed due to their sequential token-by-token prediction process.
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
As Large Language Models (LLMs) have advanced to unify understanding and generation tasks within the textual modality, recent research has shown that tasks from different modalities can also be effectively encapsulated within the NTP framework, transforming the multimodal information into tokens and predict the next one given the context.
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
With Video-MME, we extensively evaluate various state-of-the-art MLLMs, including GPT-4 series and Gemini 1. 5 Pro, as well as open-source image models like InternVL-Chat-V1. 5 and video models like LLaVA-NeXT-Video.
DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
Specifically, we trace back the semantic relevance flow from generated language tokens to raw visual encoder patches and the intermediate outputs produced by projectors.
LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation?
Diffusion models have exhibited remarkable capabilities in text-to-image generation.
Ranked #8 on
Image Captioning
on COCO Captions
(ROUGE-L metric)
Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality
Video paragraph captioning (VPC) involves generating detailed narratives for long videos, utilizing supportive modalities such as speech and event boundaries.
TempCompass: Do Video LLMs Really Understand Videos?
Motivated by these two problems, we propose the \textbf{TempCompass} benchmark, which introduces a diversity of temporal aspects and task formats.
PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain
To address this, we introduce Embodied-Instruction-Evolution (EIE), an automatic framework for synthesizing instruction tuning examples in multimodal embodied environments.
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
This work proposes TimeChat, a time-sensitive multimodal large language model specifically designed for long video understanding.
Ranked #2 on
Video-Text Retrieval
on Test-of-Time
(using extra training data)
VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models
The ability to perceive how objects change over time is a crucial ingredient in human intelligence.
FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios.
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding
TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding.
Ranked #1 on
Video Retrieval
on Condensed Movies
(using extra training data)
Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond
In this study, we explore the potential of Multimodal Large Language Models (MLLMs) in improving embodied decision-making processes for agents.
M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
To tackle this challenge and promote research in the vision-language field, we introduce the Multi-Modal, Multilingual Instruction Tuning (M$^3$IT) dataset, designed to optimize VLM alignment with human instructions.
Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition
This work proposes POMP, a prompt pre-training method for vision-language models.
Delving into the Openness of CLIP
However, evaluating the openness of CLIP-like models is challenging, as the models are open to arbitrary vocabulary in theory, but their accuracy varies in practice.
CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark
We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic.
Dynamic Knowledge Distillation for Pre-trained Language Models
Knowledge distillation~(KD) has been proved effective for compressing large-scale pre-trained language models.
Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification
Data augmentation aims to enrich training samples for alleviating the overfitting issue in low-resource or class-imbalanced situations.
Learning Relation Alignment for Calibrated Cross-modal Retrieval
To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the visual relation among the regions.
CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade
On the other hand, the exiting decisions made by internal classifiers are unreliable, leading to wrongly emitted early predictions.
DCA: Diversified Co-Attention towards Informative Live Video Commenting
In this paper, we aim to collect diversified information from video and text for informative comment generation.
Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency
Experiments on three popular datasets using convolutional as well as LSTM models show that PWWS reduces the classification accuracy to the most extent, and keeps a very low word substitution rate.
