Search Results for author:
Found 17 papers, 14 papers with code
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github. com/X-PLUG/mPLUG-Owl.
Ranked #3 on
Visual Question Answering (VQA)
on HallusionBench
Visual Question Answering (VQA)
Zero-Shot Video Question Answer
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks.
Ranked #11 on
Visual Question Answering (VQA)
on InfiMM-Eval
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
Nevertheless, without in-domain training, these models tend to ignore fine-grained OCR features, such as sophisticated tables or large blocks of text, which are essential for OCR-free document understanding.
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs.
mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model
In this work, towards a more versatile copilot for academic paper writing, we mainly focus on strengthening the multi-modal diagram analysis ability of Multimodal LLMs.
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding
In this work, we emphasize the importance of structure information in Visual Document Understanding and propose the Unified Structure Learning to boost the performance of MLLMs.
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
We further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model.
Ranked #1 on
Image Retrieval
on RUC-CAS-WenLan
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks
In addition, to facilitate a comprehensive evaluation of video-language models, we carefully build the largest human-annotated Chinese benchmarks covering three popular video-language tasks of cross-modal retrieval, video captioning, and video category classification.
Movie101: A New Movie Understanding Benchmark
Closer to real scenarios, the Movie Clip Narrating (MCN) task in our benchmark asks models to generate role-aware narration paragraphs for complete movie clips where no actors are speaking.
Question-controlled Text-aware Image Captioning
To explore how to generate personalized text-aware captions, we define a new challenging task, namely Question-controlled Text-aware Image Captioning (Qc-TextCap).
MPMQA: Multimodal Question Answering on Product Manuals
Taking into account the length of product manuals and the fact that a question is always related to a small number of pages, MPMQA can be naturally split into two subtasks: retrieving most related pages and then generating multimodal answers.
InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation
Existing metrics only provide a single score to measure caption qualities, which are less explainable and informative.
ICECAP: Information Concentrated Entity-aware Image Captioning
In this work, we focus on the entity-aware news image captioning task which aims to generate informative captions by leveraging the associated news articles to provide background knowledge about the target image.
Accommodating Audio Modality in CLIP for Multimodal Processing
In this paper, we extend the stateof-the-art Vision-Language model CLIP to accommodate the audio modality for Vision-Language-Audio multimodal processing.
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
Generalizing Multimodal Pre-training into Multilingual via Language Acquisition
Specifically, we design a lightweight language acquisition encoder based on state-of-the-art monolingual VLP models.
Explore and Tell: Embodied Visual Captioning in 3D Environments
To overcome this limitation, we propose a novel task called Embodied Captioning, which equips visual captioning models with navigation capabilities, enabling them to actively explore the scene and reduce visual ambiguity from suboptimal viewpoints.
