Search Results for author:
Found 64 papers, 43 papers with code
Can Editing LLMs Inject Harm?
Then, we find that editing attacks can inject both types of misinformation into LLMs, and the effectiveness is particularly high for commonsense misinformation injection.
RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models
Second, in cases where the model originally responds correctly, applying RAG can lead to an over-reliance on retrieved contexts, resulting in incorrect answers.
MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?
Compared with open-source VLMs, smaller-sized scoring models can provide better feedback regarding text-image alignment and image quality, while VLMs provide more accurate feedback regarding safety and generation bias due to their stronger reasoning capabilities.
CAT: Interpretable Concept-based Taylor Additive Models
Importantly, it can explain model predictions through high-level concepts that human can understand.
It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF
We validate the effectiveness of SEAM in data selection and model augmentation.
CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models
Artificial intelligence has significantly impacted medical applications, particularly with the advent of Medical Large Vision Language Models (Med-LVLMs), sparking optimism for the future of automated and personalized healthcare.
Generalizing to Unseen Domains in Diabetic Retinopathy with Disentangled Representations
Subsequently, to improve the robustness of the decoupled representations, class and domain prototypes are employed to interpolate the disentangled representations while data-aware weights are designed to focus on rare classes and domains.
Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
In this paper, we propose SIMA, a framework that enhances visual and language modality alignment through self-improvement, eliminating the needs for external models or data.
Ranked #108 on
Visual Question Answering
on MM-Vet
Calibrated Self-Rewarding Vision Language Models
In the reward modeling, we employ a step-wise strategy and incorporate visual constraints into the self-rewarding process to place greater emphasis on visual input.
Ranked #63 on
Visual Question Answering
on MM-Vet
LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models
To address these issues, we propose LITE -- a multimodal large language model for environmental ecosystems modeling.
MEIT: Multi-Modal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation
Electrocardiogram (ECG) is the primary non-invasive diagnostic tool for monitoring cardiac conditions and is crucial in assisting clinicians.
HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding
While large vision-language models (LVLMs) have demonstrated impressive capabilities in interpreting multi-modal contexts, they invariably suffer from object hallucinations (OH).
Distribution-Free Fair Federated Learning with Small Samples
As federated learning gains increasing importance in real-world applications due to its capacity for decentralized data training, addressing fairness concerns across demographic groups becomes critically important.
$C^3$: Confidence Calibration Model Cascade for Inference-Efficient Cross-Lingual Natural Language Understanding
Existing model cascade methods seek to enhance inference efficiency by greedily selecting the lightest model capable of processing the current input from a variety of models, based on model confidence scores.
Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyond
Uncertainty estimation plays a pivotal role in ensuring the reliability of safety-critical human-AI interaction systems, particularly in the medical domain.
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
This procedure is not perfect and can cause the model to hallucinate - provide answers that do not accurately reflect the image, even when the core LLM is highly factual and the vision backbone has sufficiently complete representations.
AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition
Recent advancements in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet their reliance on extensive manual labeling to provide procedural feedback remains a significant impediment.
Generating Chain-of-Thoughts with a Pairwise-Comparison Approach to Searching for the Most Promising Intermediate Thought
To improve the ability of the large language model (LLMs) to tackle complex reasoning problems, chain-of-thoughts (CoT) methods were proposed to guide LLMs to reason step-by-step, enabling problem solving from simple to complex.
Multimodal Clinical Trial Outcome Prediction with Large Language Models
To address these issues, we propose a multimodal mixture-of-experts (LIFTED) approach for clinical trial outcome prediction.
Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences
However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated.
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs
Different from prior studies, we shift our focus from evaluating standard performance to introducing a comprehensive safety evaluation suite, covering both out-of-distribution (OOD) generalization and adversarial robustness.
FREE: The Foundational Semantic Recognition for Modeling Environmental Ecosystems
This raises a fundamental question in advancing the modeling of environmental ecosystems: how to build a general framework for modeling the complex relationships amongst various environmental data over space and time?
Multimodal Representation Learning by Alternating Unimodal Adaptation
This optimization process is controlled by a gradient modification mechanism to prevent the shared head from losing previously acquired information.
Ever: Mitigating Hallucination in Large Language Models through Real-Time Verification and Rectification
Large Language Models (LLMs) have demonstrated remarkable proficiency in generating fluent text.
Fine-tuning Language Models for Factuality
The fluency and creativity of large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines.
Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges
To bridge this gap, we introduce a new benchmark, namely, the Bias and Interference Challenges in Visual Language Models (Bingo).
Conformal Prediction for Deep Classifier via Label Ranking
Conformal prediction is a statistical framework that generates prediction sets containing ground-truth labels with a desired coverage guarantee.
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
Large vision-language models (LVLMs) have shown remarkable abilities in understanding visual information with human languages.
Conservative Prediction via Data-Driven Confidence Minimization
We theoretically analyze the choice of auxiliary dataset for confidence minimization, revealing two actionable insights: (1) if the auxiliary set contains unknown examples similar to those seen at test time, confidence minimization leads to provable detection of unknown test examples, and (2) if the first condition is satisfied, it is unnecessary to filter out known examples for out-of-distribution (OOD) detection.
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions.
Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networks
As machine learning has been deployed ubiquitously across applications in modern data science, algorithmic fairness has become a great concern.
Improving Domain Generalization with Domain Relations
Distribution shift presents a significant challenge in machine learning, where models often underperform during the test stage when faced with a different distribution than the one they were trained on.
Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time
Temporal shifts -- distribution shifts arising from the passage of time -- often occur gradually and have the additional structure of timestamp metadata.
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
Multi-Domain Long-Tailed Learning by Augmenting Disentangled Representations
We study this multi-domain long-tailed learning problem and aim to produce a model that generalizes well across all classes and domains.
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the new task.
Knowledge-Driven New Drug Recommendation
However, newly approved drugs do not have much historical prescription data and cannot leverage existing drug recommendation methods.
C-Mixup: Improving Generalization in Regression
In this paper, we propose a simple yet powerful algorithm, C-Mixup, to improve generalization on regression tasks.
Spatio-Temporal Graph Few-Shot Learning with Cross-City Knowledge Transfer
To address this challenge, cross-city knowledge transfer has shown its promise, where the model learned from data-sufficient cities is leveraged to benefit the learning process of data-scarce cities.
Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation
Building models of natural language processing (NLP) is challenging in low-resource scenarios where only limited data are available.
Diversify and Disambiguate: Learning From Underspecified Data
Many datasets are underspecified: there exist multiple equally viable solutions to a given task.
Improving Out-of-Distribution Robustness via Selective Augmentation
Machine learning algorithms typically assume that training and test examples are drawn from the same distribution.
Functionally Regionalized Knowledge Transfer for Low-resource Drug Discovery
More recently, there has been a surge of interest in employing machine learning approaches to expedite the drug discovery process where virtual screening for hit discovery and ADMET prediction for lead optimization play essential roles.
Meta-learning with an Adaptive Task Scheduler
In ATS, for the first time, we design a neural scheduler to decide which meta-training tasks to use next by predicting the probability being sampled for each candidate task, and train the scheduler to optimize the generalization capacity of the meta-model to unseen tasks.
Knowledge-Aware Meta-learning for Low-Resource Text Classification
Meta-learning has achieved great success in leveraging the historical learned knowledge to facilitate the learning process of the new task.
Meta-Learning with Fewer Tasks through Task Interpolation
Meta-learning enables algorithms to quickly learn a newly encountered task with just a few labeled examples by transferring previously learned knowledge.
FEW-SHOTLEARNING WITH WEAK SUPERVISION
Few-shot meta-learning methods aim to learn the common structure shared across a set of tasks to facilitate learning new tasks with small amounts of data.
Online Structured Meta-learning
When a new task is encountered, it constructs a meta-knowledge pathway by either utilizing the most relevant knowledge blocks or exploring new blocks.
Relation-aware Meta-learning for Market Segment Demand Prediction with Limited Records
The goal is to facilitate the learning process in the target segments by leveraging the learned knowledge from data-sufficient source segments.
Improving Generalization in Meta-learning via Task Augmentation
Moreover, both MetaMix and Channel Shuffle outperform state-of-the-art results by a large margin across many datasets and are compatible with existing meta-learning algorithms.
Investigating and Mitigating Degree-Related Biases in Graph Convolutional Networks
Pseudo labels increase the chance of connecting to labeled neighbors for low-degree nodes, thus reducing the biases of GCNs from the data perspective.
Automated Relational Meta-learning
In order to efficiently learn with small amount of data on new tasks, meta-learning transfers knowledge learned from previous tasks to the new ones.
Few-Shot Knowledge Graph Completion
Knowledge graphs (KGs) serve as useful resources for various natural language processing applications.
Joint Modeling of Local and Global Temporal Dynamics for Multivariate Time Series Forecasting with Missing Values
Thus, jointly modeling local and global temporal dynamics is very promising for MTS forecasting with missing values.
Graph Few-shot Learning via Knowledge Transfer
Towards the challenging problem of semi-supervised node classification, there have been extensive studies.
Transferable Neural Processes for Hyperparameter Optimization
Automated machine learning aims to automate the whole process of machine learning, including model configuration.
Targeted Source Detection for Environmental Data
In the face of growing needs for water and energy, a fundamental understanding of the environmental impacts of human activities becomes critical for managing water and energy resources, remedying water pollution, and making regulatory policy wisely.
Transferring Robustness for Graph Neural Network Against Poisoning Attacks
To optimize PA-GNN for a poisoned graph, we design a meta-optimization algorithm that trains PA-GNN to penalize perturbations using clean graphs and their adversarial counterparts, and transfers such ability to improve the robustness of PA-GNN on the poisoned graph.
Ranked #23 on
Node Classification
on Pubmed
Hierarchically Structured Meta-learning
In order to learn quickly with few samples, meta-learning utilizes prior knowledge learned from previous tasks.
Joint Modeling of Dense and Incomplete Trajectories for Citywide Traffic Volume Inference
In this paper, we propose a novel framework for the citywide traffic volume inference using both dense GPS trajectories and incomplete trajectories captured by camera surveillance systems.
Learning from Multiple Cities: A Meta-Learning Approach for Spatial-Temporal Prediction
Specifically, our proposed model is designed as a spatial-temporal network with a meta-learning paradigm.
Revisiting Spatial-Temporal Similarity: A Deep Learning Framework for Traffic Prediction
Although both factors have been considered in modeling, existing works make strong assumptions about spatial dependence and temporal dynamics, i. e., spatial dependence is stationary in time, and temporal dynamics is strictly periodical.
Deep Multi-View Spatial-Temporal Network for Taxi Demand Prediction
Traditional demand prediction methods mostly rely on time series forecasting techniques, which fail to model the complex non-linear spatial and temporal relations.
