Search Results for author:
Found 28 papers, 20 papers with code
HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter
To address this issue, we introduce HateDay, the first global hate speech dataset representative of social media settings, randomly sampled from all tweets posted on September 21, 2022 for eight languages and four English-speaking countries.
Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation
We also show that our approach can be used for fine-grained calibration of model safety.
Compromesso! Italian Many-Shot Jailbreaks Undermine the Safety of Large Language Models
We contribute towards closing this gap by investigating the effectiveness of many-shot jailbreaking, where models are prompted with unsafe demonstrations to induce unsafe behaviour, in Italian.
Evidence of a log scaling law for political persuasion with large language models
Large language models can now generate political messages as persuasive as those written by humans, raising concerns about how far this persuasiveness may continue to increase with model size.
From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasets
We find that HS datasets for these languages exhibit a strong geo-cultural bias, largely overrepresenting a handful of countries (e. g., US and UK for English) relative to their prominence in both the broader social media population and the general population speaking these languages.
Near to Mid-term Risks and Opportunities of Open-Source Generative AI
In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education.
The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
Human feedback is central to the alignment of Large Language Models (LLMs).
Introducing v0.5 of the AI Safety Benchmark from MLCommons
We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0. 5 benchmark.
Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think
We show that the text answers are more robust to question perturbations than the first token probabilities, when the first token answers mismatch the text answers.
SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
Researchers and practitioners have met these concerns by introducing an abundance of new datasets for evaluating and improving LLM safety.
Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Dataset
Our experiments show that the resulting dataset is challenging even for state-of-the-art hate speech detection models, and that training on GAHD clearly improves model robustness.
Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ
Therefore, in this paper, we investigate the basic multilingual capabilities of state-of-the-art open LLMs beyond their intended use.
Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models
Motivated by this discrepancy, we challenge the prevailing constrained evaluation paradigm for values and opinions in LLMs and explore more realistic unconstrained evaluations.
"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models
The open-ended nature of language generation makes the evaluation of autoregressive large language models (LLMs) challenging.
SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models
While some of the models do not give a single unsafe response, most give unsafe responses to more than 20% of the prompts, with over 50% unsafe responses in the extreme.
The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
Human feedback is increasingly used to steer the behaviours of Large Language Models (LLMs).
The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models
In this paper, we address the concept of "alignment" in large language models (LLMs) through the lens of post-structuralist socio-political theory, specifically examining its parallels to empty signifiers.
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
Training large language models to follow instructions makes them perform better on a wide range of tasks and generally become more helpful.
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
In this paper, we introduce a new test suite called XSTest to identify such eXaggerated Safety behaviours in a systematic way.
The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics
To account for sociodemographics in models of individual annotator behaviour, we introduce group-specific layers to multi-annotator models.
Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback
Large language models (LLMs) are used to generate content for a wide range of tasks, and are set to reach a growing audience in coming years due to integration in product interfaces like ChatGPT or search engines like Bing.
SemEval-2023 Task 10: Explainable Detection of Online Sexism
Online sexism is a widespread and harmful phenomenon.
Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages
More data is needed, but annotating hateful content is expensive, time-consuming and potentially harmful to annotators.
Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models
To help address this issue, we introduce Multilingual HateCheck (MHC), a suite of functional tests for multilingual hate speech detection models.
Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks
To address this issue, we propose two contrasting paradigms for data annotation.
Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hate
Using the test suite, we expose weaknesses in existing hate detection models.
Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media
Token-level analysis shows that temporal adaptation captures event-driven changes in language use in the downstream task, but not those changes that are actually relevant to task performance.
HateCheck: Functional Tests for Hate Speech Detection Models
Detecting online hate is a difficult task that even state-of-the-art models struggle with.
