Search Results for author:
Found 36 papers, 27 papers with code
VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models
To analyze explicit bias, we directly pose questions to VLMs related to gender and racial differences: (1) Multiple-choice questions based on a given image (e. g., "What is the education level of the person in the image?")
VisFactor: Benchmarking Fundamental Visual Cognition in Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) have demonstrated remarkable advancements in multimodal understanding; however, their fundamental visual cognitive abilities remain largely underexplored.
Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
Multimodal Large Language Models (MLLMs) have expanded the capabilities of traditional language models by enabling interaction through both text and images.
VLMs as GeoGuessr Masters: Exceptional Performance, Hidden Biases, and Privacy Risks
Evaluating four VLMs, we find that while these models demonstrate the ability to recognize geographic information from images, achieving up to $53. 8\%$ accuracy in city prediction, they exhibit significant regional biases.
CoSER: Coordinating LLM-Based Persona Simulation of Established Roles
It provides authentic dialogues with real-world intricacies, as well as diverse data types such as conversation setups, character experiences and internal thoughts.
FairCode: Evaluating Social Bias of LLMs in Code Generation
In this study, we introduce FairCode, a novel benchmark for evaluating bias in code generation.
On the Shortcut Learning in Multilingual Neural Machine Translation
By carefully designing experiments on different MNMT scenarios and models, we attribute the off-target issue to the overfitting of the shortcuts of (non-centric, centric) language mappings.
Insight Over Sight? Exploring the Vision-Knowledge Conflicts in Multimodal LLMs
This paper explores the problem of commonsense-level vision-knowledge conflict in Multimodal Large Language Models (MLLMs), where visual information contradicts model's internal commonsense knowledge (see Figure 1).
InstantIR: Blind Image Restoration with Instant Generative Reference
In this paper, we introduce Instant-reference Image Restoration (InstantIR), a novel diffusion-based BIR method which dynamically adjusts generation condition during inference.
Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step
To assess the safety of existing models, we introduce a novel jailbreaking method called Chain-of-Jailbreak (CoJ) attack, which compromises image generation models through a step-by-step editing process.
Learning to Ask: When LLM Agents Meet Unclear Instruction
Equipped with the capability to call functions, modern large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone.
On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents
Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents, each focusing on a specific domain.
Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence.
InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context
In this paper, we developed a novel framework, InterIntent, to assess LLMs' social intelligence by mapping their ability to understand and manage intentions in a game setting.
How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO
However, existing studies focus on imitating well-known public figures or fictional characters, overlooking the potential for simulating ordinary individuals.
How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments
Researchers have examined LLMs' decision-making through the lens of Game Theory.
LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
We introduce LogicAsker, a novel approach for evaluating and enhancing the logical reasoning capabilities of large language models (LLMs) such as ChatGPT and GPT-4.
New Job, New Gender? Measuring the Social Bias in Image Generation Models
BiasPainter uses a diverse range of seed images of individuals and prompts the image generation models to edit these images using gender, race, and age-neutral queries.
The Earth is Flat? Unveiling Factual Errors in Large Language Models
Current methods for evaluating LLMs' veracity are limited by test data leakage or the need for extensive human labor, hindering efficient and accurate error detection.
Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models
Ideally, an advanced agent should possess the ability to accurately describe a given word using an aggressive description while concurrently maximizing confusion in the conservative description, enhancing its participation in the game.
InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
Then, with InCharacter, we show that state-of-the-art RPAs exhibit personalities highly aligned with the human-perceived personalities of the characters, achieving an accuracy up to 80. 7%.
Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models
This paper identifies a cultural dominance issue within large language models (LLMs) due to the predominant use of English data in model training (e. g., ChatGPT).
All Languages Matter: On the Multilingual Safety of Large Language Models
In this work, we build the first multilingual safety benchmark for LLMs, XSafety, in response to the global deployment of LLMs in practice.
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
Large Language Models (LLMs) have recently showcased their remarkable capacities, not only in natural language processing tasks but also across diverse domains such as clinical medicine, legal consultation, and education.
An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software
Moreover, through retraining the models with the test cases generated by OASIS, the robustness of the moderation model can be improved without performance degradation.
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
We propose a novel framework CipherChat to systematically examine the generalizability of safety alignment to non-natural languages -- ciphers.
Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench
Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse.
Revisiting the Reliability of Psychological Scales on Large Language Models
Recent research has focused on examining Large Language Models' (LLMs) characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics.
ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback
Therefore, we propose ParroT, a framework to enhance and regulate the translation abilities during chat based on open-source LLMs (e. g., LLaMA), human-written translation and feedback data.
Improving the Transferability of Adversarial Samples by Path-Augmented Method
However, such methods selected the image augmentation path heuristically and may augment images that are semantics-inconsistent with the target images, which harms the transferability of the generated adversarial samples.
MTTM: Metamorphic Testing for Textual Content Moderation Software
In addition, we leverage the test cases generated by MTTM to retrain the model we explored, which largely improves model robustness (0% to 5. 9% EFR) while maintaining the accuracy on the original test set.
Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine
By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e. g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages.
Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languages
This paper describes Tencent's multilingual machine translation systems for the WMT22 shared task on Large-Scale Machine Translation Evaluation for African Languages.
AEON: A Method for Automatic Evaluation of NLP Test Cases
However, in practice, many of the generated test cases fail to preserve similar semantic meaning and are unnatural (e. g., grammar errors), which leads to a high false alarm rate and unnatural test cases.
Improving Adversarial Transferability via Neuron Attribution-Based Attacks
Deep neural networks (DNNs) are known to be vulnerable to adversarial examples.
