Search Results for author:
Found 48 papers, 18 papers with code
Prescribing the Right Remedy: Mitigating Hallucinations in Large Vision-Language Models via Targeted Instruction Tuning
In this paper, we propose a targeted instruction data generation framework named DFTG that tailored to the hallucination specificity of different models.
Inference-Time Rule Eraser: Distilling and Removing Bias Rules to Mitigate Bias in Deployed Models
Fairness is critical for artificial intelligence systems, especially for those deployed in high-stakes applications such as hiring and justice.
Exploring the Privacy Protection Capabilities of Chinese Large Language Models
Large language models (LLMs), renowned for their impressive capabilities in various tasks, have significantly advanced artificial intelligence.
How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendation
Through multiple experiments on two widely-used datasets in the field of conversational recommendation, we highlight several issues with the current evaluation methods for user simulators based on LLMs: (1) Data leakage, which occurs in conversational history and the user simulator's replies, results in inflated evaluation results.
AIGCs Confuse AI Too: Investigating and Explaining Synthetic Image-induced Hallucinations in Large Vision-Language Models
Remarkably, our findings shed light on a consistent AIGC \textbf{hallucination bias}: the object hallucinations induced by synthetic images are characterized by a greater quantity and a more uniform position distribution, even these synthetic images do not manifest unrealistic or additional relevant visual features compared to natural images.
Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning
In the first phase, based on human supervision, the quality of weak supervision is enhanced through a combination of scalable oversight and ensemble learning, reducing the capability gap between weak teachers and strong students.
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
To assess the performance of Mobile-Agent, we introduced Mobile-Eval, a benchmark for evaluating mobile device operations.
Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugs
To address this issue, We propose a language-assisted diagnostic method that uses texts instead of images to diagnose bugs in vision models based on multi-modal models (eg CLIP).
CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models
This benchmark serves as a valuable resource for cultural studies in LLMs, paving the way for more culturally aware and sensitive models.
Adversarial Prompt Tuning for Vision-Language Models
With the rapid advancement of multimodal learning, pre-trained Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable capacities in bridging the gap between visual and language modalities.
AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
Despite making significant progress in multi-modal tasks, current Multi-modal Large Language Models (MLLMs) encounter the significant challenge of hallucinations, which may lead to harmful consequences.
Evaluation and Analysis of Hallucination in Large Vision-Language Models
In this paper, we propose Hallucination Evaluation based on Large Language Models (HaELM), an LLM-based hallucination evaluation framework.
Benign Shortcut for Debiasing: Fair Visual Recognition via Intervention with Shortcut Features
To this end, we propose \emph{Shortcut Debiasing}, to first transfer the target task's learning of bias attributes from bias features to shortcut features, and then employ causal intervention to eliminate shortcut features during inference.
You talk what you read: Understanding News Comment Behavior by Dispositional and Situational Attribution
Many news comment mining studies are based on the assumption that comment is explicitly linked to the corresponding news.
CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility
In this paper, we present CValues, the first Chinese human values evaluation benchmark to measure the alignment ability of LLMs in terms of both safety and responsibility criteria.
Introducing Foundation Models as Surrogate Models: Advancing Towards More Practical Adversarial Attacks
Harnessing the concept of non-robust features, we elaborate on two guiding principles for surrogate model selection to explain why the foundational model is an optimal choice for this role.
Towards Alleviating the Object Bias in Prompt Tuning-based Factual Knowledge Extraction
Many works employed prompt tuning methods to automatically optimize prompt queries and extract the factual knowledge stored in Pretrained Language Models.
Towards Black-box Adversarial Example Detection: A Data Reconstruction-based Method
Black-box attack, which is a more realistic threat and has led to various black-box adversarial training-based defense methods, however, does not attract considerable attention in adversarial example detection.
Echoes: Unsupervised Debiasing via Pseudo-bias Labeling in an Echo Chamber
This paper first presents experimental analyses revealing that the existing biased models overfit to bias-conflicting samples in the training data, which negatively impacts the debiasing performance of the target models.
From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping
Although previous works have created generation capacity for CLIP through additional language models, a modality gap between the CLIP representations of different modalities and the inability of CLIP to model the offset of this gap, which fails the concept to transfer across modalities.
Improved Visual Fine-tuning with Natural Language Supervision
Fine-tuning a visual pre-trained model can leverage the semantic information from large-scale pre-training data and mitigate the over-fitting problem on downstream vision tasks with limited training examples.
Backdoor for Debias: Mitigating Model Bias with Backdoor Attack-based Artificial Bias
In this work, we found that the backdoor attack can construct an artificial bias similar to the model bias derived in standard training.
Unlearnable Clusters: Towards Label-agnostic Unlearnable Examples
Furthermore, we propose to leverage VisionandLanguage Pre-trained Models (VLPMs) like CLIP as the surrogate model to improve the transferability of the crafted UCs to diverse domains.
Artificial Intelligence Security Competition (AISC)
The security of artificial intelligence (AI) is an important research area towards safe, reliable, and trustworthy AI systems.
Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment
We further propose Anchor Augment to guide the generative model's attention to the fine-grained information in the representation of CLIP.
Fair Visual Recognition via Intervention with Proxy Features
To this end, we propose \emph{Proxy Debiasing}, to first transfer the target task's learning of bias information from bias features to artificial proxy features, and then employ causal intervention to eliminate proxy features in inference.
FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralization
Although FairCLIP is used to eliminate bias in image retrieval, it achieves the neutralization of the representation which is common to all CLIP downstream tasks.
Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models
Vision-Language Pre-training (VLP) models have achieved state-of-the-art performance in numerous cross-modal tasks.
Towards Adversarial Attack on Vision-Language Pre-training Models
While vision-language pre-training model (VLP) has shown revolutionary improvements on various vision-language (V+L) tasks, the studies regarding its adversarial robustness remain largely unexplored.
Low-Mid Adversarial Perturbation against Unauthorized Face Recognition System
In light of the growing concerns regarding the unauthorized use of facial recognition systems and its implications on individual privacy, the exploration of adversarial perturbations as a potential countermeasure has gained traction.
Investigating and Explaining the Frequency Bias in Image Classification
Specifically, our investigations verify that the spectral density of datasets mainly affects the learning priority, while the class consistency mainly affects the feature discrimination.
Non-generative Generalized Zero-shot Learning via Task-correlated Disentanglement and Controllable Samples Synthesis
Synthesizing pseudo samples is currently the most effective way to solve the Generalized Zero-Shot Learning (GZSL) problem.
Debiasing Backdoor Attack: A Benign Application of Backdoor Attack in Eliminating Data Bias
Backdoor attack is a new AI security risk that has emerged in recent years.
Understanding and Testing Generalization of Deep Networks on Out-of-Distribution Data
Deep network models perform excellently on In-Distribution (ID) data, but can significantly fail on Out-Of-Distribution (OOD) data.
Learning to Learn a Cold-start Sequential Recommender
The cold-start recommendation is an urgent problem in contemporary online applications.
Knowledge Graph-enhanced Sampling for Conversational Recommender System
Then, two samplers are designed to enhance knowledge by sampling fuzzy samples with high uncertainty for obtaining user preferences and reliable negative samples for updating recommender to achieve efficient acquisition of user preferences and model updating, and thus provide a powerful solution for CRS to deal with E&E problem.
Towards Predictable Feature Attribution: Revisiting and Improving Guided BackPropagation
Recently, backpropagation(BP)-based feature attribution methods have been widely adopted to interpret the internal mechanisms of convolutional neural networks (CNNs), and expected to be human-understandable (lucidity) and faithful to decision-making processes (fidelity).
Benign Adversarial Attack: Tricking Models for Goodness
In spite of the successful application in many fields, machine learning models today suffer from notorious problems like vulnerability to adversarial examples.
ImageNet Pre-training also Transfers Non-Robustness
ImageNet pre-training has enabled state-of-the-art results on many tasks.
An Experimental Study of Semantic Continuity for Deep Learning Models
Deep learning models suffer from the problem of semantic discontinuity: small perturbations in the input space tend to cause semantic-level interference to the model output.
Towards Accuracy-Fairness Paradox: Adversarial Example-based Data Augmentation for Visual Debiasing
Our data analysis on facial attribute recognition demonstrates (1) the attribution of model bias from imbalanced training data distribution and (2) the potential of adversarial examples in balancing data distribution.
Adversarial Privacy-preserving Filter
While widely adopted in practical applications, face recognition has been critically discussed regarding the malicious use of face images and the potential privacy problems, e. g., deceiving payment system and causing personal sabotage.
MMCGAN: Generative Adversarial Network with Explicit Manifold Prior
Since the basic assumption of conventional manifold learning fails in case of sparse and uneven data distribution, we introduce a new target, Minimum Manifold Coding (MMC), for manifold learning to encourage simple and unfolded manifold.
Adaptive Adversarial Logits Pairing
Specifically, AALP consists of an adaptive feature optimization module with Guided Dropout to systematically pursue fewer high-contribution features, and an adaptive sample weighting module by setting sample-specific training weights to balance between logits pairing loss and classification loss.
A Generalization Theory based on Independent and Task-Identically Distributed Assumption
The derived generalization bound based on the ITID assumption identifies the significance of hypothesis invariance in guaranteeing generalization performance.
blessing in disguise: Designing Robust Turing Test by Employing Algorithm Unrobustness
Turing test was originally proposed to examine whether machine's behavior is indistinguishable from a human.
Attention, Please! Adversarial Defense via Activation Rectification and Preservation
This study provides a new understanding of the adversarial attack problem by examining the correlation between adversarial attack and visual attention change.
Understanding Deep Learning Generalization by Maximum Entropy
DNN is then regarded as approximating the feature conditions with multilayer feature learning, and proved to be a recursive solution towards maximum entropy principle.
