Search Results for author:
Found 50 papers, 43 papers with code
GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling
We evaluate the GDI-Bench on various open-source and closed-source models, conducting decoupled analyses in the visual and reasoning domains.
MM-IFEngine: Towards Multimodal Instruction Following
The Instruction Following (IF) ability measures how well Multi-modal Large Language Models (MLLMs) understand exactly what users are telling them and whether they are doing it right.
Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing
To address this gap, we introduce RISEBench, the first benchmark for evaluating Reasoning-Informed viSual Editing (RISE).
LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?
Multi-step spatial reasoning entails understanding and reasoning about spatial relationships across multiple sequential steps, which is crucial for tackling complex real-world applications, such as robotic manipulation, autonomous navigation, and automated assembly.
Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM
To address this gap, we introduce Creation-MMBench, a multimodal benchmark specifically designed to evaluate the creative capabilities of MLLMs in real-world, image-based tasks.
Information Density Principle for MLLM Benchmarks
Therefore, we propose a critical principle of Information Density, which examines how much insight a benchmark can provide for the development of MLLMs.
Image Quality Assessment: From Human to Machine Preference
Considering the huge gap between human and machine visual systems, this paper proposes the topic: Image Quality Assessment for Machine Vision for the first time.
VisualPRM: An Effective Process Reward Model for Multimodal Reasoning
We introduce VisualPRM, an advanced multimodal Process Reward Model (PRM) with 8B parameters, which improves the reasoning abilities of existing Multimodal Large Language Models (MLLMs) across different model scales and families with Best-of-N (BoN) evaluation strategies.
Visual-RFT: Visual Reinforcement Fine-Tuning
Reinforcement Fine-Tuning (RFT) in Large Reasoning Models like OpenAI o1 learns from feedback on its answers, which is especially useful in applications when fine-tuning data is scarce.
Few-Shot Object Detection
Fine-Grained Image Classification
+4
OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference
Recent advancements in open-source multi-modal large language models (MLLMs) have primarily focused on enhancing foundational capabilities, leaving a significant gap in human preference alignment.
VideoRoPE: What Makes for Good Video Rotary Position Embedding?
As part of our analysis, we introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors) task, which adds periodic distractors into V-NIAH.
Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement
The quality of Supervised Fine-Tuning (SFT) data plays a critical role in enhancing the conversational capabilities of Large Language Models (LLMs).
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs.
OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?
OVO-Bench evaluates the ability of video LLMs to reason and respond to events occurring at specific timestamps under three distinct scenarios: (1) Backward tracing: trace back to past events to answer the question.
BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning
To this end, we focus on improving the reasoning quality within each step and present BoostStep.
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
Recent advancements in multimodal large language models (MLLMs) have made significant strides in open-world understanding.
MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
As a prominent direction of Artificial General Intelligence (AGI), Multimodal Large Language Models (MLLMs) have garnered increased attention from both industry and academia.
MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
Existing visual alignment methods, primarily designed for single-image scenarios, struggle to effectively handle the complexity of multi-image tasks due to the scarcity of diverse training data and the high cost of annotating chosen/rejected pairs.
CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution
In this report, we introduce \textbf{CompassJudger-1}, the first open-source \textbf{all-in-one} judge LLM.
ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
To address these shortcomings, we introduce ProSA, a framework designed to evaluate and comprehend prompt sensitivity in LLMs.
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields.
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
Based on the evaluation results obtained with the toolkit, we host OpenVLM Leaderboard, a comprehensive leaderboard to track the progress of multi-modality learning research.
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
This long-context capability allows IXC-2. 5 to excel in tasks requiring extensive input and output contexts.
Ranked #4 on
Video Question Answering
on TVBench
MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
We propose the integration of an additional high-resolution visual encoder to capture fine-grained details, which are then fused with base visual features through a Conv-Gate fusion network.
Ranked #75 on
Visual Question Answering
on MM-Vet
Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Vision Language Models (VLMs) demonstrate remarkable proficiency in addressing a wide array of visual questions, which requires strong perception and reasoning faculties.
MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
The advent of large vision-language models (LVLMs) has spurred research into their applications in multi-modal contexts, particularly in video understanding.
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
To this end, we meticulously designed a differential video captioning strategy, which is stable, scalable, and efficient for generating captions for videos with arbitrary resolution, aspect ratios, and length.
MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark
MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills.
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution.
Ranked #47 on
Visual Question Answering
on MM-Vet
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks
Recently, the large language model (LLM) community has shown increasing interest in enhancing LLMs' capability to handle extremely long documents.
Are We on the Right Way for Evaluating Large Vision-Language Models?
We evaluate 16 leading LVLMs on MMStar to assess their multi-modal capabilities, and on 7 benchmarks with the proposed metrics to investigate their data leakage and actual multi-modal gain.
InternLM2 Technical Report
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI).
Ranked #5 on
Long-Context Understanding
on Ada-LEval (BestAnswer)
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension.
Ranked #18 on
Visual Question Answering
on MM-Vet v2
BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues
In contrast, other LLMs struggle to generate multi-turn dialogues of satisfactory quality due to poor instruction-following capability, tendency to generate lengthy utterances, or limited general capability.
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition
We propose InternLM-XComposer, a vision-language large model that enables advanced image-text comprehension and composition.
Ranked #9 on
Visual Question Answering (VQA)
on InfiMM-Eval
SkeleTR: Towrads Skeleton-based Action Recognition in the Wild
It first models the intra-person skeleton dynamics for each skeleton sequence with graph convolutions, and then uses stacked Transformer encoders to capture person interactions that are important for action recognition in general scenarios.
MMBench: Is Your Multi-modal Model an All-around Player?
In response to these challenges, we propose MMBench, a bilingual benchmark for assessing the multi-modal capabilities of VLMs.
Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
Specifically, we construct a negative-sample-free triplet steam structure that is composed of an anchor stream without any masking, a spatial masking stream with Central Spatial Masking (CSM), and a temporal masking stream with Motion Attention Temporal Masking (MATM).
SkeleTR: Towards Skeleton-based Action Recognition in the Wild
It first models the intra-person skeleton dynamics for each skeleton sequence with graph convolutions, and then uses stacked Transformer encoders to capture person interactions that are important for action recognition in the wild.
Ranked #3 on
Human Interaction Recognition
on NTU RGB+D
DG-STGCN: Dynamic Spatial-Temporal Modeling for Skeleton-based Action Recognition
Graph convolution networks (GCN) have been widely used in skeleton-based action recognition.
Ranked #14 on
Skeleton Based Action Recognition
on NTU RGB+D
Mitigating Representation Bias in Action Recognition: Algorithms and Benchmarks
Deep learning models have achieved excellent recognition results on large-scale video benchmarks.
PYSKL: Towards Good Practices for Skeleton Action Recognition
The toolbox supports a wide variety of skeleton action recognition algorithms, including approaches based on GCN and CNN.
Ranked #28 on
Skeleton Based Action Recognition
on NTU RGB+D 120
TransRank: Self-supervised Video Representation Learning via Ranking-based Transformation Recognition
To mitigate this problem, we developed TransRank, a unified framework for recognizing Transformations in a Ranking formulation.
OCSampler: Compressing Videos to One Clip with Single-step Sampling
Recent works prefer to formulate frame sampling as a sequential decision task by selecting frames one by one according to their importance, while we present a new paradigm of learning instance-specific video condensation policies to select informative frames for representing the entire video only in a single step.
Revisiting Skeleton-based Action Recognition
In this work, we propose PoseC3D, a new approach to skeleton-based action recognition, which relies on a 3D heatmap stack instead of a graph sequence as the base representation of human skeletons.
Ranked #1 on
Action Recognition
on Volleyball
Omni-sourced Webly-supervised Learning for Video Recognition
Then a joint-training strategy is proposed to deal with the domain gaps between multiple data sources and formats in webly-supervised learning.
Ranked #7 on
Action Recognition
on UCF101
(using extra training data)
TRB: A Novel Triplet Representation for Understanding 2D Human Body
In this paper, we propose the Triplet Representation for Body (TRB) -- a compact 2D human body representation, with skeleton keypoints capturing human pose information and contour keypoints containing human shape information.
Conditional Image Generation
Open-Ended Question Answering
+1
SRPGAN: Perceptual Generative Adversarial Network for Single Image Super Resolution
In this paper, we build a super resolution perceptual generative adversarial network (SRPGAN) framework for SISR tasks.
