Search Results for author: Max Bartolo

Found 16 papers, 7 papers with code

Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation

no code implementations14 Feb 2024 Jessica Quaye, Alicia Parrish, Oana Inel, Charvi Rastogi, Hannah Rose Kirk, Minsuk Kahng, Erin Van Liemt, Max Bartolo, Jess Tsang, Justin White, Nathan Clement, Rafael Mosquera, Juan Ciro, Vijay Janapa Reddi, Lora Aroyo

By focusing on ``implicitly adversarial'' prompts (those that trigger T2I models to generate unsafe images for non-obvious reasons), we isolate a set of difficult safety issues that human creativity is well-suited to uncover.

Text-to-Image Generation

Human Feedback is not Gold Standard

1 code implementation28 Sep 2023 Tom Hosking, Phil Blunsom, Max Bartolo

We critically analyse the use of human feedback for both training and evaluation, to verify whether it fully captures a range of crucial error criteria.

Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

2 code implementations CVPR 2022 Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, Candace Ross

We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground.

Visual Reasoning

Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks

1 code implementation ACL 2022 Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, Douwe Kiela

We introduce Dynatask: an open source system for setting up custom NLP tasks that aims to greatly lower the technical knowledge and effort required for hosting and evaluating state-of-the-art NLP models, as well as for conducting model in the loop data collection with crowdworkers.

Benchmarking

Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants

no code implementations NAACL 2022 Max Bartolo, Tristan Thrush, Sebastian Riedel, Pontus Stenetorp, Robin Jia, Douwe Kiela

We collect training datasets in twenty experimental settings and perform a detailed analysis of this approach for the task of extractive question answering (QA) for both standard and adversarial data collection.

Extractive Question-Answering Question Answering

Contrasting Human- and Machine-Generated Word-Level Adversarial Examples for Text Classification

1 code implementation EMNLP 2021 Maximilian Mozes, Max Bartolo, Pontus Stenetorp, Bennett Kleinberg, Lewis D. Griffin

Research shows that natural language processing models are generally considered to be vulnerable to adversarial attacks; but recent work has drawn attention to the issue of validating these adversarial inputs against certain criteria (e. g., the preservation of semantics and grammaticality).

Sentiment Analysis Sentiment Classification +3

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

1 code implementation ACL 2022 Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, Pontus Stenetorp

When primed with only a handful of training samples, very large, pretrained language models such as GPT-3 have shown competitive results when compared to fully-supervised, fine-tuned, large, pretrained language models.

text-classification Text Classification

Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation

no code implementations EMNLP 2021 Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, Douwe Kiela

We further conduct a novel human-in-the-loop evaluation to show that our models are considerably more robust to new human-written adversarial examples: crowdworkers can fool our model only 8. 8% of the time on average, compared to 17. 6% for a model trained without synthetic data.

Answer Selection Question Generation

Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension

1 code implementation2 Feb 2020 Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, Pontus Stenetorp

We find that training on adversarially collected samples leads to strong generalisation to non-adversarially collected datasets, yet with progressive performance deterioration with increasingly stronger models-in-the-loop.

 Ranked #1 on Reading Comprehension on AdversarialQA (using extra training data)

Reading Comprehension

Cannot find the paper you are looking for? You can Submit a new open access paper.