Search Results for author:
Found 18 papers, 7 papers with code
SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models
In this paper, we present and delve into the self-consistency capability of LVLMs, a crucial aspect that reflects the models' ability to both generate informative captions for specific objects and subsequently utilize these captions to accurately re-identify the objects in a closed-loop process.
VL-Mamba: Exploring State Space Models for Multimodal Learning
The extensive experiments on diverse multimodal benchmarks with competitive performance show the effectiveness of our proposed VL-Mamba and demonstrate the great potential of applying state space models for multimodal learning tasks.
Ranked #61 on
Visual Question Answering
on MM-Vet
Knowledge Condensation and Reasoning for Knowledge-based VQA
We condense the retrieved knowledge passages from two perspectives.
Unveiling Parts Beyond Objects:Towards Finer-Granularity Referring Expression Segmentation
To foster future research into fine-grained visual grounding, our benchmark RefCOCOm, the MRES-32M dataset and model UniRES will be publicly available at https://github. com/Rubics-Xuan/MRES
EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE
Owing to the combination of the unified architecture and pre-training task, EVE is easy to scale up, enabling better downstream performance with fewer resources and faster training speed.
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
We show that only language-paired two-modality data is sufficient to connect all modalities.
Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner
In this paper, we propose a novel method called Joint QA and DC GEneration (JADE), which utilizes a pre-trained multimodal model and easily-crawled image-text pairs to automatically generate and filter large-scale VQA and dense captioning datasets.
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner.
Ranked #1 on
Video Captioning
on VATEX
(using extra training data)
MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learning
Our method performs joint masking on image-text input and integrates both implicit and explicit targets for the masked signals to recover.
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources.
Ranked #1 on
Image Retrieval
on Localized Narratives
CPTR: Full Transformer Network for Image Captioning
Besides, we provide detailed visualizations of the self-attention between patches in the encoder and the "words-to-patches" attention in the decoder thanks to the full Transformer architecture.
Fast Sequence Generation with Multi-Agent Reinforcement Learning
These models are autoregressive in that they generate each word by conditioning on previously generated words, which leads to heavy latency during inference.
AutoCaption: Image Captioning with Neural Architecture Search
The whole process involves a visual understanding module and a language generation module, which brings more challenges to the design of deep neural networks than other tasks.
Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning
In this paper, we propose a Non-Autoregressive Image Captioning (NAIC) model with a novel training paradigm: Counterfactuals-critical Multi-Agent Learning (CMAL).
Normalized and Geometry-Aware Self-Attention Network for Image Captioning
First, we propose Normalized Self-Attention (NSA), a reparameterization of SA that brings the benefits of normalization inside SA.
Vatex Video Captioning Challenge 2020: Multi-View Features and Hybrid Reward Strategies for Video Captioning
This report describes our solution for the VATEX Captioning Challenge 2020, which requires generating descriptions for the videos in both English and Chinese languages.
Aligning Linguistic Words and Visual Semantic Units for Image Captioning
Image captioning attempts to generate a sentence composed of several linguistic words, which are used to describe objects, attributes, and interactions in an image, denoted as visual semantic units in this paper.
MSCap: Multi-Style Image Captioning With Unpaired Stylized Text
The discriminator and the generator are trained in an adversarial manner to enable more natural and human-like captions.
