Search Results for author: Adina Williams

Found 49 papers, 28 papers with code

Sometimes We Want Ungrammatical Translations

1 code implementation Findings (EMNLP) 2021 Prasanna Parthasarathi, Koustuv Sinha, Joelle Pineau, Adina Williams

Rapid progress in Neural Machine Translation (NMT) systems over the last few years has focused primarily on improving translation quality, and as a secondary focus, improving robustness to perturbations (e. g. spelling).

Machine Translation NMT +1

Measuring the Similarity of Grammatical Gender Systems by Comparing Partitions

no code implementations EMNLP 2020 Arya D. McCarthy, Adina Williams, Shijia Liu, David Yarowsky, Ryan Cotterell

Of particular interest, languages on the same branch of our phylogenetic tree are notably similar, whereas languages from separate branches are no more similar than chance.

Community Detection

Generalising to German Plural Noun Classes, from the Perspective of a Recurrent Neural Network

1 code implementation CoNLL (EMNLP) 2021 Verna Dankers, Anna Langedijk, Kate McCurdy, Adina Williams, Dieuwke Hupkes

Inflectional morphology has since long been a useful testing ground for broader questions about generalisation in language and the viability of neural network models as cognitive models of language.

Grammatical Gender's Influence on Distributional Semantics: A Causal Perspective

no code implementations30 Nov 2023 Karolina Stańczak, Kevin Du, Adina Williams, Isabelle Augenstein, Ryan Cotterell

However, when we control for the meaning of the noun, we find that grammatical gender has a near-zero effect on adjective choice, thereby calling the neo-Whorfian hypothesis into question.

ROBBIE: Robust Bias Evaluation of Large Generative Language Models

no code implementations29 Nov 2023 David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, Eric Michael Smith

In this work, our focus is two-fold: (1) Benchmarking: a comparison of 6 different prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative LLMs.

Benchmarking Fairness

The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks

1 code implementation26 Oct 2023 Kaiser Sun, Adina Williams, Dieuwke Hupkes

NLP models have progressed drastically in recent years, according to numerous datasets proposed to evaluate performance.

DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity

no code implementations11 Aug 2023 Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, Adriana Romero Soriano

The unprecedented photorealistic results achieved by recent text-to-image generative systems and their increasing use as plug-and-play content creation solutions make it crucial to understand their potential biases.

Benchmarking Image Generation

Weisfeiler and Lehman Go Measurement Modeling: Probing the Validity of the WL Test

1 code implementation11 Jul 2023 Arjun Subramonian, Adina Williams, Maximilian Nickel, Yizhou Sun, Levent Sagun

The expressive power of graph neural networks is usually measured by comparing how many pairs of graphs or nodes an architecture can possibly distinguish as non-isomorphic to those distinguishable by the $k$-dimensional Weisfeiler-Lehman ($k$-WL) test.


Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus

1 code implementation27 Jan 2023 Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, Chengxu Zhuang

In partnership with CoNLL and CMCL, we provide a platform for approaches to pretraining with a limited-size corpus sourced from data inspired by the input to children.

Language Acquisition Language Modelling +1

Language model acceptability judgements are not always robust to context

no code implementations18 Dec 2022 Koustuv Sinha, Jon Gauthier, Aaron Mueller, Kanishka Misra, Keren Fuentes, Roger Levy, Adina Williams

In this paper, we investigate the stability of language models' performance on targeted syntactic evaluations as we vary properties of the input context: the length of the context, the types of syntactic phenomena it contains, and whether or not there are violations of grammaticality.

Language Modelling

The Curious Case of Absolute Position Embeddings

1 code implementation23 Oct 2022 Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, Adina Williams

Transformer language models encode the notion of word order using positional information.

Perturbation Augmentation for Fairer NLP

1 code implementation25 May 2022 Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, Adina Williams

Unwanted and often harmful social biases are becoming ever more salient in NLP research, affecting both models and datasets.


"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset

2 code implementations18 May 2022 Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, Adina Williams

As language models grow in popularity, it becomes increasingly important to clearly measure all possible markers of demographic identity in order to avoid perpetuating existing societal harms.

Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

2 code implementations CVPR 2022 Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, Candace Ross

We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground.

Visual Reasoning

Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks

1 code implementation ACL 2022 Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, Douwe Kiela

We introduce Dynatask: an open source system for setting up custom NLP tasks that aims to greatly lower the technical knowledge and effort required for hosting and evaluating state-of-the-art NLP models, as well as for conducting model in the loop data collection with crowdworkers.


A Latent-Variable Model for Intrinsic Probing

2 code implementations20 Jan 2022 Karolina Stańczak, Lucas Torroba Hennigen, Adina Williams, Ryan Cotterell, Isabelle Augenstein

The success of pre-trained contextualized representations has prompted researchers to analyze them for the presence of linguistic information.

Analyzing Dynamic Adversarial Training Data in the Limit

1 code implementation Findings (ACL) 2022 Eric Wallace, Adina Williams, Robin Jia, Douwe Kiela

To create models that are robust across a wide range of test inputs, training datasets should include diverse examples that span numerous phenomena.


Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking

no code implementations NeurIPS 2021 Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, Douwe Kiela

We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform.


Investigating Failures of Automatic Translation in the Case of Unambiguous Gender

no code implementations ACL 2022 Adithya Renduchintala, Adina Williams

Transformer based models are the modern work horses for neural machine translation (NMT), reaching state of the art across several benchmarks.

Machine Translation NMT +1

Sometimes We Want Translationese

no code implementations15 Apr 2021 Prasanna Parthasarathi, Koustuv Sinha, Joelle Pineau, Adina Williams

Rapid progress in Neural Machine Translation (NMT) systems over the last few years has been driven primarily towards improving translation quality, and as a secondary focus, improved robustness to input perturbations (e. g. spelling and grammatical mistakes).

Machine Translation NMT +1

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

no code implementations EMNLP 2021 Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela

A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines.

Language Modelling Masked Language Modeling

UnNatural Language Inference

1 code implementation ACL 2021 Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, Adina Williams

We provide novel evidence that complicates this claim: we find that state-of-the-art Natural Language Inference (NLI) models assign the same labels to permuted examples as they do to the original, i. e. they are largely invariant to random word-order permutations.

Natural Language Inference Natural Language Understanding

To what extent do human explanations of model behavior align with actual model behavior?

no code implementations EMNLP (BlackboxNLP) 2021 Grusha Prasad, Yixin Nie, Mohit Bansal, Robin Jia, Douwe Kiela, Adina Williams

Given the increasingly prominent role NLP models (will) play in our lives, it is important for human expectations of model behavior to align with actual model behavior.

Natural Language Inference

ANLIzing the Adversarial Natural Language Inference Dataset

1 code implementation SCiL 2022 Adina Williams, Tristan Thrush, Douwe Kiela

We perform an in-depth error analysis of Adversarial NLI (ANLI), a recently introduced large-scale human-and-model-in-the-loop natural language inference dataset collected over multiple rounds.

Natural Language Inference

Intrinsic Probing through Dimension Selection

1 code implementation EMNLP 2020 Lucas Torroba Hennigen, Adina Williams, Ryan Cotterell

Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks.

Word Embeddings

Pareto Probing: Trading Off Accuracy for Complexity

1 code implementation EMNLP 2020 Tiago Pimentel, Naomi Saphra, Adina Williams, Ryan Cotterell

In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume.

Dependency Parsing

A Tale of a Probe and a Parser

1 code implementation ACL 2020 Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, Ryan Cotterell

One such probe is the structural probe (Hewitt and Manning, 2019), designed to quantify the extent to which syntactic information is encoded in contextualised word representations.

Contextualised Word Representations

On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs

no code implementations3 May 2020 Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damián Blasi, Hanna Wallach

We also find that there are statistically significant relationships between the grammatical genders of inanimate nouns and the verbs that take those nouns as direct objects, as indirect objects, and as subjects.

Predicting Declension Class from Form and Meaning

1 code implementation ACL 2020 Adina Williams, Tiago Pimentel, Arya D. McCarthy, Hagen Blix, Eleanor Chodroff, Ryan Cotterell

We find for two Indo-European languages (Czech and German) that form and meaning respectively share significant amounts of information with class (and contribute additional information above and beyond gender).

Multi-Dimensional Gender Bias Classification

no code implementations EMNLP 2020 Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, Adina Williams

We show our classifiers prove valuable for a variety of important applications, such as controlling for gender bias in generative models, detecting gender bias in arbitrary text, and shed light on offensive language in terms of genderedness.

Classification General Classification

Information-Theoretic Probing for Linguistic Structure

1 code implementation ACL 2020 Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, Ryan Cotterell

The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually ``know'' about natural language.

Word Embeddings

Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition

1 code implementation ACL 2020 Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, Adina Williams

We use IMPPRES to evaluate whether BERT, InferSent, and BOW NLI models trained on MultiNLI (Williams et al., 2018) learn to make pragmatic inferences.

Implicatures Natural Language Inference +1

Adversarial NLI: A New Benchmark for Natural Language Understanding

2 code implementations ACL 2020 Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, Douwe Kiela

We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure.

Natural Language Understanding Test

Quantifying the Semantic Core of Gender Systems

no code implementations IJCNLP 2019 Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damián Blasi, Hanna Wallach

To that end, we use canonical correlation analysis to correlate the grammatical gender of inanimate nouns with an externally grounded definition of their lexical semantics.

On the Idiosyncrasies of the Mandarin Chinese Classifier System

no code implementations NAACL 2019 Shijia Liu, Hongyuan Mei, Adina Williams, Ryan Cotterell

While idiosyncrasies of the Chinese classifier system have been a richly studied topic among linguists (Adams and Conklin, 1973; Erbaugh, 1986; Lakoff, 1986), not much work has been done to quantify them with statistical methods.

Verb Argument Structure Alternations in Word and Sentence Embeddings

no code implementations WS 2019 Katharina Kann, Alex Warstadt, Adina Williams, Samuel R. Bowman

For converging evidence, we further construct LaVA, a corresponding word-level dataset, and investigate whether the same syntactic features can be extracted from word embeddings.

Sentence Embedding Sentence-Embedding +2

Do latent tree learning models identify meaningful structure in sentences?

1 code implementation TACL 2018 Adina Williams, Andrew Drozdov, Samuel R. Bowman

Recent work on the problem of latent tree learning has made it possible to train neural networks that learn to both parse a sentence and use the resulting parse to interpret the sentence, all without exposure to ground-truth parse trees at training time.

Sentence Classification

The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations

no code implementations WS 2017 Nikita Nangia, Adina Williams, Angeliki Lazaridou, Samuel R. Bowman

This paper presents the results of the RepEval 2017 Shared Task, which evaluated neural network sentence representation learning models on the Multi-Genre Natural Language Inference corpus (MultiNLI) recently introduced by Williams et al. (2017).

Natural Language Inference Representation Learning +1

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

3 code implementations NAACL 2018 Adina Williams, Nikita Nangia, Samuel R. Bowman

This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding.

BIG-bench Machine Learning Domain Adaptation +1

Cannot find the paper you are looking for? You can Submit a new open access paper.