Search Results for author:
Found 26 papers, 20 papers with code
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Attempting to complement this deficiency, we investigate layerwise properties of LoRA on fine-tuning tasks and observe an uncommon skewness of weight norms across different layers.
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself.
Ranked #29 on
Visual Question Answering
on MM-Vet
The Instinctive Bias: Spurious Images lead to Hallucination in MLLMs
In this paper, we identify a typical class of inputs that baffles MLLMs, which consist of images that are highly relevant but inconsistent with answers, causing MLLMs to suffer from hallucination.
SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark
The paper introduces SceMQA, a novel benchmark for scientific multimodal question answering at the college entrance level.
PipeNet: Question Answering with Semantic Pruning over Knowledge Graphs
To facilitate the evaluation of pruned subgraphs, we also propose a graph attention network (GAT) based module to reason with the subgraph data.
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
The deployment of multimodal large language models (MLLMs) has brought forth a unique vulnerability: susceptibility to malicious attacks through visual inputs.
Exploring Boundary of GPT-4V on Marine Analysis: A Preliminary Case Study
In this study, we carry out the preliminary and comprehensive case study of utilizing GPT-4V for marine analysis.
G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model
We first analyze the limitations of current Multimodal Large Language Models (MLLMs) in this area: they struggle to accurately comprehending basic geometric elements and their relationships.
Plum: Prompt Learning using Metaheuristic
Since the emergence of large language models, prompt learning has become a popular method for optimizing and customizing these models.
PerceptionGPT: Effectively Fusing Visual Perception into LLM
In this paper, we present a novel end-to-end framework named PerceptionGPT, which efficiently and effectively equips the VLLMs with visual perception abilities by leveraging the representation power of LLMs' token embedding.
MarineGPT: Unlocking Secrets of Ocean to the Public
Large language models (LLMs), such as ChatGPT/GPT-4, have proven to be powerful tools in promoting the user experience as an AI assistant.
Non-Autoregressive Sentence Ordering
Existing sentence ordering approaches generally employ encoder-decoder frameworks with the pointer net to recover the coherence by recurrently predicting each sentence step-by-step.
Mitigating the Alignment Tax of RLHF
Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different reward-tax trade-offs, we propose Adaptive Model Averaging (AMA) to adaptively find various combination ratios of model layers.
tdCoxSNN: Time-Dependent Cox Survival Neural Network for Continuous-time Dynamic Prediction
In pursuit of constructing a dynamic prediction model for a progressive eye disorder, age-related macular degeneration (AMD), we propose a time-dependent Cox survival neural network (tdCoxSNN) to predict its progression using longitudinal fundus images.
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models
As the number of available models and specialized tasks keeps growing, the job of general finetuning becomes highly nontrivial.
DetGPT: Detect What You Need via Reasoning
Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines.
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
Utilizing a reward model and a sufficient number of samples, our approach selects the high-quality samples, discarding those that exhibit undesired behavior, and subsequently enhancing the model by fine-tuning on these filtered samples.
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks
X-FM has one language encoder, one vision encoder, and one fusion encoder, as well as a new training method.
Ranked #3 on
Visual Grounding
on RefCOCO+ test B
Generalizing Math Word Problem Solvers via Solution Diversification
In this paper, we design a new training framework for an MWP solver by introducing a solution buffer and a solution discriminator.
Analogical Math Word Problems Solving with Enhanced Problem-Solution Association
In this paper, we propose to build a novel MWP solver by leveraging analogical MWPs, which advance the solver's generalization ability across different kinds of MWPs.
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
Vision language pre-training aims to learn alignments between vision and language from a large amount of data.
Ranked #1 on
Cross-Modal Retrieval
on Flickr30k
(using extra training data)
Execution-based Evaluation for Data Science Code Generation Models
Code generation models can benefit data scientists' productivity by automatically generating code from context and text descriptions.
MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving
Math word problem (MWP) solving faces a dilemma in number representation learning.
Ranked #5 on
Math Word Problem Solving
on MathQA
Graph-to-Tree Learning for Solving Math Word Problems
While the recent tree-based neural models have demonstrated promising results in generating solution expression for the math word problem (MWP), most of these models do not capture the relationships and order information among the quantities well.
Ranked #10 on
Math Word Problem Solving
on Math23K
Template-based math word problem solvers with recursive neural networks
Then, we design a recursive neural network to encode the quantity with Bi-LSTM and self attention, and infer the unknown operator nodes in a bottom-up manner.
Modeling Intra-Relation in Math Word Problems with Different Functional Multi-Head Attentions
Several deep learning models have been proposed for solving math word problems (MWPs) automatically.
Ranked #13 on
Math Word Problem Solving
on Math23K
