Search Results for author:
Found 20 papers, 15 papers with code
How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments
Additionally, we conduct evaluations across various LLMs and find that GPT-4 outperforms other models on GAMA-Bench, achieving a score of 72. 5.
New Job, New Gender? Measuring the Social Bias in Image Generation Models
BiasPainter uses a diverse range of seed images of individuals and prompts the image generation models to edit these images using gender, race, and age-neutral queries.
The Earth is Flat? Unveiling Factual Errors in Large Language Models
Current methods for evaluating LLMs' veracity are limited by test data leakage or the need for extensive human labor, hindering efficient and accurate error detection.
A & B == B & A: Triggering Logical Reasoning Failures in Large Language Models
In addition, the test cases of LogicAsker can be further used to design demonstration examples for in-context learning, which effectively improves the logical reasoning ability of LLMs, e. g., 10\% for GPT-4.
Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models
Ideally, an advanced agent should possess the ability to accurately describe a given word using an aggressive description while concurrently maximizing confusion in the conservative description, enhancing its participation in the game.
InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales.
Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models
This paper identifies a cultural dominance issue within large language models (LLMs) due to the predominant use of English data in model training (e. g., ChatGPT).
All Languages Matter: On the Multilingual Safety of Large Language Models
In this work, we build the first multilingual safety benchmark for LLMs, XSafety, in response to the global deployment of LLMs in practice.
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
Large Language Models (LLMs) have recently showcased their remarkable capacities, not only in natural language processing tasks but also across diverse domains such as clinical medicine, legal consultation, and education.
An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software
Moreover, through retraining the models with the test cases generated by OASIS, the robustness of the moderation model can be improved without performance degradation.
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
We propose a novel framework CipherChat to systematically examine the generalizability of safety alignment to non-natural languages -- ciphers.
Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench
Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse.
Revisiting the Reliability of Psychological Scales on Large Language Models
Recent research has extended beyond assessing the performance of Large Language Models (LLMs) to examining their characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics.
ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback
Therefore, we propose ParroT, a framework to enhance and regulate the translation abilities during chat based on open-source LLMs (e. g., LLaMA), human-written translation and feedback data.
Improving the Transferability of Adversarial Samples by Path-Augmented Method
However, such methods selected the image augmentation path heuristically and may augment images that are semantics-inconsistent with the target images, which harms the transferability of the generated adversarial samples.
MTTM: Metamorphic Testing for Textual Content Moderation Software
In addition, we leverage the test cases generated by MTTM to retrain the model we explored, which largely improves model robustness (0% to 5. 9% EFR) while maintaining the accuracy on the original test set.
Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine
By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e. g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages.
Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languages
This paper describes Tencent's multilingual machine translation systems for the WMT22 shared task on Large-Scale Machine Translation Evaluation for African Languages.
AEON: A Method for Automatic Evaluation of NLP Test Cases
However, in practice, many of the generated test cases fail to preserve similar semantic meaning and are unnatural (e. g., grammar errors), which leads to a high false alarm rate and unnatural test cases.
Improving Adversarial Transferability via Neuron Attribution-Based Attacks
Deep neural networks (DNNs) are known to be vulnerable to adversarial examples.
