Search Results for author:
Found 12 papers, 3 papers with code
MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model
This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning.
MIBench: Evaluating Multimodal Large Language Models over Multiple Images
However, most existing MLLMs and benchmarks primarily focus on single-image input scenarios, leaving the performance of MLLMs when handling realistic multiple images remain underexplored.
Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models
Large Vision Language Models exhibit remarkable capabilities but struggle with hallucinations inconsistencies between images and their descriptions.
Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection
Vision Transformers (ViTs) have become increasingly popular in large-scale Vision and Language Pre-training (VLP) models.
TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training
Self-supervised Multi-modal Contrastive Learning (SMCL) remarkably advances modern Vision-Language Pre-training (VLP) models by aligning visual and linguistic modalities.
Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
We first analyzed the representation distribution of textual and visual tokens in MLLM, revealing two important findings: 1) there is a significant gap between textual and visual representations, indicating unsatisfactory cross-modal representation alignment; 2) representations of texts that contain and do not contain hallucinations are entangled, making it challenging to distinguish them.
Ranked #116 on
Visual Question Answering
on MM-Vet
BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization
Specifically, We incorporate a Text-Semantics-Aware Patch Selector (TSPS) into the ViT backbone to perform a coarse-grained visual token extraction and then attach a flexible Transformer-based Patch Abstraction Decoder (PAD) upon the backbone for top-level visual abstraction.
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
To ensure the reliability of PandaLM, we collect a diverse human-annotated test dataset, where all contexts are generated by humans and labels are aligned with human preferences.
Exploiting Pseudo Image Captions for Multimodal Summarization
Cross-modal contrastive learning in vision language pretraining (VLP) faces the challenge of (partial) false negatives.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Cross-modal contrastive learning in vision language pretraining (VLP) faces the challenge of (partial) false negatives.
BUS: Efficient and Effective Vision-Language Pre-Training with Bottom-Up Patch Summarization.
In this paper, we propose a Bottom-Up Patch Summarization approach named BUS which is inspired by the Document Summarization Task in NLP to learn a concise visual summary of lengthy visual token sequences, guided by textual semantics.
SIMILARITY LEARNING FOR COVER SONG IDENTIFICATION USING CROSS-SIMILARITY MATRICES OF MULTI-LEVEL DEEP SEQUENCES
One part is a network for learn- ing the deep sequence representation of music tracks, and the other is a similarity estimation network which takes as input the cross- similarity matrices calculated from the deep sequences of a pair of tracks.
Ranked #1 on
Cover song identification
on YouTube350
