no code implementations • SIGDIAL (ACL) 2022 • A. Stevie Bergman, Gavin Abercrombie, Shannon Spruit, Dirk Hovy, Emily Dinan, Y-Lan Boureau, Verena Rieser
Over the last several years, end-to-end neural conversational agents have vastly improved their ability to carry unrestricted, open-domain conversations with humans.
no code implementations • EMNLP 2021 • Amanda Cercas Curry, Gavin Abercrombie, Verena Rieser
We find that the distribution of abuse is vastly different compared to other commonly used datasets, with more sexually tinted aggression towards the virtual persona of these systems.
no code implementations • ACL 2022 • Emily Dinan, Gavin Abercrombie, A. Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, Verena Rieser
We then empirically assess the extent to which current tools can measure these effects and current systems display them.
no code implementations • 4 Oct 2024 • Aiqi Jiang, Nikolas Vitsakis, Tanvi Dinkar, Gavin Abercrombie, Ioannis Konstas
Gender-Based Violence (GBV) is an increasing problem online, but existing datasets fail to capture the plurality of possible annotator perspectives or ensure the representation of affected groups.
no code implementations • 1 Jul 2024 • Gavin Abercrombie, Djalel Benbouzid, Paolo Giudici, Delaram Golpayegani, Julio Hernandez, Pierre Noro, Harshvardhan Pandit, Eva Paraschou, Charlie Pownall, Jyoti Prajapati, Mark A. Sayre, Ushnish Sengupta, Arthit Suriyawongkul, Ruby Thelot, Sofia Vei, Laura Waltersdorfer
This paper introduces a collaborative, human-centred taxonomy of AI, algorithmic and automation harms.
no code implementations • 29 Mar 2024 • Helena Bonaldi, Yi-Ling Chung, Gavin Abercrombie, Marco Guerini
In recent years, counterspeech has emerged as one of the most promising strategies to fight online hate.
1 code implementation • 5 Mar 2024 • Flor Miriam Plaza-del-Arco, Amanda Cercas Curry, Alba Curry, Gavin Abercrombie, Dirk Hovy
We then analyze the emotions generated by the models in relation to the gender-event pairs.
no code implementations • 4 Mar 2024 • Amanda Cercas Curry, Gavin Abercrombie, Zeerak Talat
Natural language processing research has begun to embrace the notion of annotator subjectivity, motivated by variations in labelling.
no code implementations • 1 Jul 2023 • Yi-Ling Chung, Gavin Abercrombie, Florence Enock, Jonathan Bright, Verena Rieser
Counterspeech offers direct rebuttals to hateful speech by challenging perpetrators of hate and showing support to targets of abuse.
no code implementations • 16 May 2023 • Gavin Abercrombie, Amanda Cercas Curry, Tanvi Dinkar, Verena Rieser, Zeerak Talat
In this paper, we discuss the linguistic factors that contribute to the anthropomorphism of dialogue systems and the harms that can arise, including reinforcing gender stereotypes and notions of acceptable language.
no code implementations • 16 May 2023 • Fatma Elsafoury, Gavin Abercrombie
In this paper, we trace the biases in current natural language processing (NLP) models back to their origins in racism, sexism, and homophobia over the last 500 years.
no code implementations • 10 May 2023 • Nikolas Vitsakis, Amit Parekh, Tanvi Dinkar, Gavin Abercrombie, Ioannis Konstas, Verena Rieser
There are two competing approaches for modelling annotator disagreement: distributional soft-labelling approaches (which aim to capture the level of disagreement) or modelling perspectives of individual annotators or groups thereof.
no code implementations • 2 May 2023 • Anya Belz, Craig Thomson, Ehud Reiter, Gavin Abercrombie, Jose M. Alonso-Moral, Mohammad Arvan, Anouck Braggaar, Mark Cieliebak, Elizabeth Clark, Kees Van Deemter, Tanvi Dinkar, Ondřej Dušek, Steffen Eger, Qixiang Fang, Mingqi Gao, Albert Gatt, Dimitra Gkatzia, Javier González-Corbelle, Dirk Hovy, Manuela Hürlimann, Takumi Ito, John D. Kelleher, Filip Klubicka, Emiel Krahmer, Huiyuan Lai, Chris van der Lee, Yiru Li, Saad Mahamood, Margot Mieskes, Emiel van Miltenburg, Pablo Mosteiro, Malvina Nissim, Natalie Parde, Ondřej Plátek, Verena Rieser, Jie Ruan, Joel Tetreault, Antonio Toral, Xiaojun Wan, Leo Wanner, Lewis Watson, Diyi Yang
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible.
no code implementations • 28 Apr 2023 • Elisa Leonardelli, Alexandra Uma, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Massimo Poesio
We report on the second LeWiDi shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on NLP, instead of both NLP and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements-as training with aggregated labels for subjective NLP tasks is a particularly obvious misrepresentation of the data; and (iii) for the evaluation, we concentrate on soft approaches to evaluation.
no code implementations • 25 Jan 2023 • Gavin Abercrombie, Verena Rieser, Dirk Hovy
We commonly use agreement measures to assess the utility of judgements made by human annotators in Natural Language Processing (NLP) tasks.
1 code implementation • 2 Oct 2022 • Gavin Abercrombie, Verena Rieser
While individual crowdworkers may be unreliable at grading the seriousness of the prompts, their aggregated labels tend to agree with professional opinion to a greater extent on identifying the medical queries and recognising the risk types posed by the responses.
1 code implementation • 20 Sep 2021 • Amanda Cercas Curry, Gavin Abercrombie, Verena Rieser
We find that the distribution of abuse is vastly different compared to other commonly used datasets, with more sexually tinted aggression towards the virtual persona of these systems.
no code implementations • 7 Jul 2021 • Emily Dinan, Gavin Abercrombie, A. Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, Verena Rieser
Over the last several years, end-to-end neural conversational agents have vastly improved in their ability to carry a chit-chat conversation with humans.
1 code implementation • ACL (GeBNLP) 2021 • Gavin Abercrombie, Amanda Cercas Curry, Mugdha Pandya, Verena Rieser
Technology companies have produced varied responses to concerns about the effects of the design of their conversational AI systems.
no code implementations • LREC 2020 • Gavin Abercrombie, Riza Batista-Navarro
These include a linear classifier as well as a neural network trained using a transformer word embedding model (BERT), and fine-tuned on the parliamentary speeches.
no code implementations • CONLL 2019 • Gavin Abercrombie, Federico Nanni, Riza Batista-Navarro, Simone Paolo Ponzetto
Debate motions (proposals) tabled in the UK Parliament contain information about the stated policy preferences of the Members of Parliament who propose them, and are key to the analysis of all subsequent speeches given in response to them.
no code implementations • WS 2019 • Gavin Abercrombie, Riza Batista-Navarro
We investigate changes in the meanings of words used in the UK Parliament across two different epochs.
no code implementations • 9 Jul 2019 • Gavin Abercrombie, Riza Batista-Navarro
In this article we present the results of a systematic literature review of 61 studies, all of which address the automatic analysis of the sentiment and opinions expressed and positions taken by speakers in parliamentary (and other legislative) debates.
no code implementations • WS 2018 • Gavin Abercrombie, Riza Theresa Batista-Navarro
Analysis of the topics mentioned and opinions expressed in parliamentary debate motions{--}or proposals{--}is difficult for human readers, but necessary for understanding and automatic processing of the content of the subsequent speeches.
no code implementations • LREC 2016 • Gavin Abercrombie
By concentrating on translation for assimilation (gist comprehension) from Scots to English, it is proposed that the development of dictionaries designed to be used with in the Apertium platform will be sufficient to produce translations that improve non-Scots speakers understanding of the language.