Search Results for author:
Found 16 papers, 12 papers with code
Process Reinforcement through Implicit Rewards
While dense rewards also offer an appealing choice for the reinforcement learning (RL) of LLMs since their fine-grained rewards have the potential to address some inherent issues of outcome rewards, such as training efficiency and credit assignment, this potential remains largely unrealized.
EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
To address this, we propose EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs with embodied tasks.
Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions
For instance, Euclid outperforms the best closed-source model, Gemini-1. 5-Pro, by up to 58. 56% on certain Geoperception benchmark tasks and 10. 65% on average across all tasks.
A Topic-level Self-Correctional Approach to Mitigate Hallucinations in MLLMs
Through a deconfounded strategy that replaces each topic within the response with the best or worst alternatives generated by the model itself, TPO creates more contrasting pairwise preference feedback, enhancing the feedback quality without human or proprietary model intervention.
MiniCPM-V: A GPT-4V Level MLLM on Your Phone
The recent surge of Multimodal Large Language Models (MLLMs) has fundamentally reshaped the landscape of AI research and industry, shedding light on a promising path toward the next AI milestone.
Ranked #7 on
Multiple-choice
on Neptune-Full
RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness
Traditional feedback learning for hallucination reduction relies on labor-intensive manual labeling or expensive proprietary models.
Ranked #1 on
Visual Question Answering
on AMBER
UltraWiki: Ultra-fine-grained Entity Set Expansion with Negative Seed Entities
To solve this issue, we first introduce negative seed entities in the inputs, which belong to the same fine-grained semantic class as the positive seed entities but differ in certain attributes.
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction.
Ranked #1 on
Visual Question Answering
on VQA v2
Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
The capabilities of MLLMs depend on two crucial factors: the model architecture to facilitate the feature alignment of visual modules and large language models; the multimodal instruction tuning datasets for human instruction following.
Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Building a competitive counterpart in other languages is highly challenging due to the low-resource nature of non-English multimodal data (i. e., lack of large-scale, high-quality image-text data).
SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding
However, LLMs are sometimes too footloose for natural language understanding (NLU) tasks which always have restricted output and input format.
MESED: A Multi-modal Entity Set Expansion Dataset with Fine-grained Semantic Classes and Hard Negative Entities
The Entity Set Expansion (ESE) task aims to expand a handful of seed entities with new entities belonging to the same semantic class.
Knowledge-augmented Few-shot Visual Relation Detection
Inspired by this, we devise a knowledge-augmented, few-shot VRD framework leveraging both textual knowledge and visual relation knowledge to improve the generalization ability of few-shot VRD.
Visually Grounded Commonsense Knowledge Acquisition
In this work, we present CLEVER, which formulates CKE as a distantly supervised multi-instance learning problem, where models learn to summarize commonsense relations from a bag of images about an entity pair without any human annotation on image instances.
Embracing Ambiguity: Improving Similarity-oriented Tasks with Contextual Synonym Knowledge
Extensive experiments demonstrate that PICSO can dramatically outperform the original PLMs and the other knowledge and synonym injection models on four different similarity-oriented tasks.
Contrastive Learning with Hard Negative Entities for Entity Set Expansion
In addition, we propose the ProbExpan, a novel probabilistic ESE framework utilizing the entity representation obtained by the aforementioned language model to expand entities.
