Search Results for author: Scott A. Hale

Found 26 papers, 5 papers with code

SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models

no code implementations14 Nov 2023 Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk, Rebecca Qian, Anand Kannappan, Scott A. Hale, Paul Röttger

While some of the models do not give a single unsafe response, most give unsafe responses to more than 20% of the prompts, with over 50% unsafe responses in the extreme.


Lost in Translation -- Multilingual Misinformation and its Evolution

no code implementations27 Oct 2023 Dorian Quelle, Calvin Cheng, Alexandre Bovet, Scott A. Hale

Using fact-checks as a proxy for the spread of misinformation, we find 33% of repeated claims cross linguistic boundaries, suggesting that some misinformation permeates language barriers.

Fact Checking Misinformation +3

The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models

no code implementations3 Oct 2023 Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, Scott A. Hale

In this paper, we address the concept of "alignment" in large language models (LLMs) through the lens of post-structuralist socio-political theory, specifically examining its parallels to empty signifiers.

Casteist but Not Racist? Quantifying Disparities in Large Language Model Bias between India and the West

no code implementations15 Sep 2023 Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, Scott A. Hale

In this paper, we quantify stereotypical bias in popular LLMs according to an Indian-centric frame and compare bias levels between the Indian and Western contexts.

Language Modelling Large Language Model

DoDo Learning: DOmain-DemOgraphic Transfer in Language Models for Detecting Abuse Targeted at Public Figures

1 code implementation31 Jul 2023 Hannah Rose Kirk, Angus R. Williams, Liam Burke, Yi-Ling Chung, Ivan Debono, Pica Johansson, Francesca Stevens, Jonathan Bright, Scott A. Hale

We find that (i) small amounts of diverse data are hugely beneficial to generalisation and model adaptation; (ii) models transfer more easily across demographics but models trained on cross-domain data are more generalisable; (iii) some groups contribute more to generalisability than others; and (iv) dataset similarity is a signal of transferability.

text-classification Text Classification

Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback

no code implementations9 Mar 2023 Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, Scott A. Hale

Large language models (LLMs) are used to generate content for a wide range of tasks, and are set to reach a growing audience in coming years due to integration in product interfaces like ChatGPT or search engines like Bing.

Query Rewriting for Effective Misinformation Discovery

no code implementations14 Oct 2022 Ashkan Kazemi, Artem Abzaliev, Naihao Deng, Rui Hou, Scott A. Hale, Verónica Pérez-Rosas, Rada Mihalcea

We propose a novel system to help fact-checkers formulate search queries for known misinformation claims and effectively search across multiple social media platforms.

Misinformation reinforcement-learning +2

Top Gear or Black Mirror: Inferring Political Leaning From Non-Political Content

no code implementations11 Aug 2022 Ahmet Kurnaz, Scott A. Hale

Polarization and echo chambers are often studied in the context of explicitly political events such as elections, and little scholarship has examined the mixing of political groups in non-political contexts.

Matching Tweets With Applicable Fact-Checks Across Languages

no code implementations14 Feb 2022 Ashkan Kazemi, Zehua Li, Verónica Pérez-Rosas, Scott A. Hale, Rada Mihalcea

We conduct both classification and retrieval experiments, in monolingual (English only), multilingual (Spanish, Portuguese), and cross-lingual (Hindi-English) settings using multilingual transformer models such as XLM-RoBERTa and multilingual embeddings such as LaBSE and SBERT.

Fact Checking Retrieval

Fairness via AI: Bias Reduction in Medical Information

no code implementations6 Sep 2021 Shiri Dori-Hacohen, Roberto Montenegro, Fabricio Murai, Scott A. Hale, Keen Sung, Michela Blain, Jennifer Edwards-Johnson

While part (3) of this work specifically focuses on the health domain, the fundamental computer science advances and contributions stemming from research efforts in bias reduction and Fairness via AI have broad implications in all areas of society.

Bias Detection Fairness +3

Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal Hate

no code implementations Findings (ACL) 2021 Austin Botelho, Bertie Vidgen, Scott A. Hale

We show that both text- and visual- enrichment improves model performance, with the multimodal model (0. 771) outperforming other models' F1 scores (0. 544, 0. 737, and 0. 754).

Claim Matching Beyond English to Scale Global Fact-Checking

no code implementations ACL 2021 Ashkan Kazemi, Kiran Garimella, Devin Gaffney, Scott A. Hale

We train our own embedding model using knowledge distillation and a high-quality "teacher" model in order to address the imbalance in embedding quality between the low- and high-resource languages in our dataset.

Fact Checking Knowledge Distillation

Demographic Inference and Representative Population Estimates from Multilingual Social Media Data

no code implementations15 May 2019 Zijian Wang, Scott A. Hale, David Adelani, Przemyslaw A. Grabowicz, Timo Hartmann, Fabian Flöck, David Jurgens

In a large experiment over multilingual heterogeneous European regions, we show that our demographic inference and bias correction together allow for more accurate estimates of populations and make a significant step towards representative social sensing in downstream applications with multilingual social media.


Measuring the Volatility of the Political agenda in Public Opinion and News Media

1 code implementation27 Aug 2018 Chico Q. Camargo, Scott A. Hale, Peter John, Helen Z. Margetts

Recent election surprises, regime changes, and political shocks indicate that political agendas have become more fast-moving and volatile.

Foreign-language Reviews: Help or Hindrance?

no code implementations1 Feb 2017 Scott A. Hale, Irene Eleta

The number and quality of user reviews greatly affects consumer purchasing decisions.

User Reviews and Language: How Language Influences Ratings

no code implementations6 May 2016 Scott A. Hale

The number of user reviews of tourist attractions, restaurants, mobile apps, etc.

Understanding Editing Behaviors in Multilingual Wikipedia

no code implementations28 Aug 2015 Suin Kim, Sungjoon Park, Scott A. Hale, Sooyoung Kim, Jeongmin Byun, Alice Oh

We study multilingualism by collecting and analyzing a large dataset of the content written by multilingual editors of the English, German, and Spanish editions of Wikipedia.

How much is said in a microblog? A multilingual inquiry based on Weibo and Twitter

no code implementations1 Jun 2015 Han-Teng Liao, King-wa Fu, Scott A. Hale

This paper presents a multilingual study on, per single post of microblog text, (a) how much can be said, (b) how much is written in terms of characters and bytes, and (c) how much is said in terms of information content in posts by different organizations in different languages.

Cross-language Wikipedia Editing of Okinawa, Japan

no code implementations4 Jan 2015 Scott A. Hale

This article analyzes users who edit Wikipedia articles about Okinawa, Japan, in English and Japanese.

Multilinguals and Wikipedia Editing

no code implementations3 Dec 2013 Scott A. Hale

This article analyzes one month of edits to Wikipedia in order to examine the role of users editing multiple language editions (referred to as multilingual users).

Cannot find the paper you are looking for? You can Submit a new open access paper.