Search Results for author:
Found 46 papers, 29 papers with code
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
Based on the evaluation results obtained with the toolkit, we host OpenVLM Leaderboard, a comprehensive leaderboard to track the progress of multi-modality learning research.
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
This long-context capability allows IXC-2. 5 to excel in tasks requiring extensive input and output contexts.
Ranked #25 on
Visual Question Answering
on MM-Vet
MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations
Moreover, 33. 2% of the questions are cross-page questions requiring evidence across multiple pages.
MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
Generating natural and meaningful responses to communicate with multi-modal human inputs is a fundamental capability of Large Vision-Language Models(LVLMs).
Ranked #60 on
Visual Question Answering
on MM-Vet
V3Det Challenge 2024 on Vast Vocabulary and Open Vocabulary Object Detection: Methods and Results
2) Open Vocabulary Object Detection: This track goes a step further, requiring algorithms to detect objects from an open set of categories, including unknown objects.
MotionClone: Training-Free Motion Cloning for Controllable Video Generation
Motion-based controllable text-to-video generation involves motions to control the video generation.
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
To this end, we meticulously designed a differential video captioning strategy, which is stable, scalable, and efficient for generating captions for videos with arbitrary resolution, aspect ratios, and length.
Bootstrap3D: Improving 3D Content Creation with Synthetic Data
Leveraging this pipeline, we have generated 1 million high-quality synthetic multi-view images with dense descriptive captions to address the shortage of high-quality 3D data.
Streaming Long Video Understanding with Large Language Models
After the encoding process, the Adaptive Memory Selection strategy selects a constant number of question-related memories from all the historical memories and feeds them into the LLM to generate informative responses.
ReasonPix2Pix: Instruction Reasoning Dataset for Advanced Image Editing
Instruction-based image editing focuses on equipping a generative model with the capacity to adhere to human-written instructions for editing images.
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
Compared to both open-source and proprietary models, InternVL 1. 5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks.
Ranked #10 on
Visual Question Answering
on MM-Vet v2
Unified Scene Representation and Reconstruction for 3D Large Language Models
Existing approaches extract point clouds either from ground truth (GT) geometry or 3D scenes reconstructed by auxiliary models.
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution.
Ranked #19 on
Visual Question Answering
on MM-Vet
Are We on the Right Way for Evaluating Large Vision-Language Models?
We evaluate 16 leading LVLMs on MMStar to assess their multi-modal capabilities, and on 7 benchmarks with the proposed metrics to investigate their data leakage and actual multi-modal gain.
InternLM2 Technical Report
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI).
Ranked #5 on
Long-Context Understanding
on Ada-LEval (BestAnswer)
Long-CLIP: Unlocking the Long-Text Capability of CLIP
Contrastive Language-Image Pre-training (CLIP) has been the cornerstone for zero-shot classification, text-image retrieval, and text-image generation by aligning image and text modalities.
RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition
Notably, our approach demonstrates a significant improvement in performance on 5 fine-grained visual recognition benchmarks, 11 few-shot image recognition datasets, and the 2 object detection datasets under the zero-shot recognition setting.
SongComposer: A Large Language Model for Lyric and Melody Composition in Song Generation
We present SongComposer, an innovative LLM designed for song composition.
DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models
We present DualFocus, a novel framework for integrating macro and micro perspectives within multi-modal large language models (MLLMs) to enhance vision-language task performance.
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension.
Ranked #14 on
Visual Question Answering
on MM-Vet v2
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
Based on the observation, OPERA introduces a penalty term on the model logits during the beam-search decoding to mitigate the over-trust issue, along with a rollback strategy that retrospects the presence of summary tokens in the previously generated tokens, and re-allocate the token selection if necessary.
Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
Multimodal large language models have made significant advancements in recent years, yet they still suffer from a common issue known as the "hallucination problem", in which the models generate textual descriptions that inaccurately depict or entirely fabricate content from associated images.
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
In the realm of large multi-modal models (LMMs), efficient modality alignment is crucial yet often constrained by the scarcity of high-quality image-text data.
Ranked #2 on
visual instruction following
on LLaVA-Bench
PersonMAE: Person Re-Identification Pre-Training with Masked AutoEncoders
Pre-training is playing an increasingly important role in learning generic feature representation for Person Re-identification (ReID).
Emotional Listener Portrait: Neural Listener Head Generation with Emotion
Listener head generation centers on generating non-verbal behaviors (e. g., smile) of a listener in reference to the information delivered by a speaker.
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition
We propose InternLM-XComposer, a vision-language large model that enables advanced image-text comprehension and composition.
Ranked #9 on
Visual Question Answering (VQA)
on InfiMM-Eval
MLLM-DataEngine: An Iterative Refinement Approach for MLLM
Despite the great advance of Multimodal Large Language Models (MLLMs) in both instruction dataset building and benchmarking, the independence of training and evaluation makes current MLLMs hard to further improve their capability under the guidance of evaluation results with a relatively low human cost.
VIGC: Visual Instruction Generation and Correction
A practical solution to this problem would be to utilize the available multimodal large language models (MLLMs) to generate instruction data for vision-language tasks.
Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting
Based on our analysis, we provide a simple yet effective way to boost the adversarial robustness of MAE.
Diversity-Aware Meta Visual Prompting
We present Diversity-Aware Meta Visual Prompting~(DAM-VP), an efficient and effective prompting method for transferring pre-trained models to downstream tasks with frozen backbone.
CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet
Recent studies have shown that CLIP has achieved remarkable success in performing zero-shot inference while its fine-tuning performance is not satisfactory.
PointCAT: Contrastive Adversarial Training for Robust Point Cloud Recognition
Notwithstanding the prominent performance achieved in various applications, point cloud recognition models have often suffered from natural corruptions and adversarial perturbations.
MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
Second, masked self-distillation is also consistent with vision-language contrastive from the perspective of training objective as both utilize the visual encoder for feature aligning, and thus is able to learn local semantics getting indirect supervision from the language.
Bootstrapped Masked Autoencoders for Vision BERT Pretraining
The first design is motivated by the observation that using a pretrained MAE to extract the features as the BERT prediction target for masked tokens can achieve better pretraining performance.
Shape-invariant 3D Adversarial Point Clouds
In this paper, we propose a novel Point-Cloud Sensitivity Map to boost both the efficiency and imperceptibility of point perturbations.
Protecting Celebrities from DeepFake with Identity Consistency Transformer
In this work we propose Identity Consistency Transformer, a novel face forgery detection method that focuses on high-level semantics, specifically identity information, and detecting a suspect face by finding identity inconsistency in inner and outer face regions.
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
This paper explores a better prediction target for BERT pre-training of vision transformers.
Mobile-Former: Bridging MobileNet and Transformer
This structure leverages the advantages of MobileNet at local processing and transformer at global interaction.
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
By further pretraining on the larger dataset ImageNet-21K, we achieve 87. 5% Top-1 accuracy on ImageNet-1K and high segmentation performance on ADE20K with 55. 7 mIoU.
Ranked #25 on
Semantic Segmentation
on ADE20K val
Identity-Driven DeepFake Detection
Our approach takes as input the suspect image/video as well as the target identity information (a reference image or video).
LG-GAN: Label Guided Adversarial Network for Flexible Targeted Attack of Point Cloud-based Deep Networks
To overcome these shortcomings, this paper proposes a novel label guided adversarial network (LG-GAN) for real-time flexible targeted point cloud attack.
GreedyFool: Distortion-Aware Sparse Adversarial Attack
Sparse adversarial samples are a special branch of adversarial samples that can fool the target model by only perturbing a few pixels.
Robust Superpixel-Guided Attentional Adversarial Attack
Firstly, because of the contradiction between the local smoothness of natural images and the noisy property of these adversarial perturbations, this "pixel-wise" way makes these methods not robust to image processing based defense methods and steganalysis based detection methods.
Self-Robust 3D Point Recognition via Gather-Vector Guidance
In this paper, we look into the problem of 3D adversary attack, and propose to leverage the internal properties of the point clouds and the adversarial examples to design a new self-robust deep neural network (DNN) based 3D recognition systems.
Once a MAN: Towards Multi-Target Attack via Learning Multi-Target Adversarial Network Once
Recently, generation-based methods have received much attention since they directly use feed-forward networks to generate the adversarial samples, which avoid the time-consuming iterative attacking procedure in optimization-based and gradient-based methods.
CAAD 2018: Powerful None-Access Black-Box Attack Based on Adversarial Transformation Network
In this paper, we propose an improvement of Adversarial Transformation Networks(ATN) to generate adversarial examples, which can fool white-box models and black-box models with a state of the art performance and won the 2rd place in the non-target task in CAAD 2018.
