Search Results for author: Tal Linzen

Found 54 papers, 25 papers with code

When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it

1 code implementation6 May 2022 Sebastian Schuster, Tal Linzen

We use this evaluation suite for a fine-grained investigation of the entity tracking abilities of the Transformer-based models GPT-2 and GPT-3.

Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models

1 code implementation Findings (ACL) 2022 Aaron Mueller, Robert Frank, Tal Linzen, Luheng Wang, Sebastian Schuster

We find that pre-trained seq2seq models generalize hierarchically when performing syntactic transformations, whereas models trained from scratch on syntactic transformations do not.

Short-term memory in neural language models

no code implementations29 Sep 2021 Kristijan Armeni, Christopher Honey, Tal Linzen

Thus, although the transformer and LSTM architectures were both trained to predict language sequences, only the transformer learned to flexibly index prior tokens.

Language Modelling

NOPE: A Corpus of Naturally-Occurring Presuppositions in English

1 code implementation CoNLL (EMNLP) 2021 Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bowman, Tal Linzen

Understanding language requires grasping not only the overtly stated content, but also making inferences about things that were left unsaid.

Frequency Effects on Syntactic Rule Learning in Transformers

1 code implementation EMNLP 2021 Jason Wei, Dan Garrette, Tal Linzen, Ellie Pavlick

Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules.

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models

1 code implementation ACL 2021 Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, Yonatan Belinkov

Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts.

Evaluating Groundedness in Dialogue Systems: The BEGIN Benchmark

no code implementations30 Apr 2021 Nouha Dziri, Hannah Rashkin, Tal Linzen, David Reitter

To facilitate evaluation of such metrics, we introduce the Benchmark for Evaluation of Grounded INteraction (BEGIN).

Language Modelling Natural Language Inference

Does Putting a Linguist in the Loop Improve NLU Data Collection?

no code implementations Findings (EMNLP) 2021 Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alex Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, Samuel R. Bowman

We take natural language inference as a test case and ask whether it is beneficial to put a linguist `in the loop' during data collection to dynamically identify and address gaps in the data by introducing novel constraints on the task.

Natural Language Inference

Information-theoretic Probing Explains Reliance on Spurious Features

no code implementations ICLR 2021 Charles Lovering, Rohan Jha, Tal Linzen, Ellie Pavlick

In this work, we test the hypothesis that the extent to which a feature influences a model's decisions can be predicted using a combination of two factors: The feature's "extractability" after pre-training (measured using information-theoretic probing techniques), and the "evidence" available during fine-tuning (defined as the feature's co-occurrence rate with the label).

COGS: A Compositional Generalization Challenge Based on Semantic Interpretation

1 code implementation EMNLP 2020 Najoung Kim, Tal Linzen

Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts.

Semantic Parsing

Universal linguistic inductive biases via meta-learning

1 code implementation29 Jun 2020 R. Thomas McCoy, Erin Grant, Paul Smolensky, Thomas L. Griffiths, Tal Linzen

To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases.

Language Acquisition Meta-Learning

How Can We Accelerate Progress Towards Human-like Linguistic Generalization?

no code implementations ACL 2020 Tal Linzen

This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has become a central tool for measuring progress in natural language understanding.

Natural Language Understanding Transfer Learning

Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs

1 code implementation ACL 2020 Michael A. Lepori, Tal Linzen, R. Thomas McCoy

Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks.

Data Augmentation

Syntactic Data Augmentation Increases Robustness to Inference Heuristics

1 code implementation ACL 2020 Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, Tal Linzen

Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets.

Data Augmentation Natural Language Inference

Syntactic Structure from Deep Learning

no code implementations22 Apr 2020 Tal Linzen, Marco Baroni

Modern deep neural networks achieve impressive performance in engineering applications that require extensive linguistic skills, such as machine translation.

Language Acquisition Machine Translation +1

Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks

no code implementations TACL 2020 R. Thomas McCoy, Robert Frank, Tal Linzen

We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection.

Quantity doesn't buy quality syntax with neural language models

no code implementations IJCNLP 2019 Marten van Schijndel, Aaron Mueller, Tal Linzen

We investigate to what extent these shortcomings can be mitigated by increasing the size of the network and the corpus on which it is trained.

RNNs implicitly implement tensor-product representations

1 code implementation ICLR 2019 R. Thomas McCoy, Tal Linzen, Ewan Dunbar, Paul Smolensky

Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).

Representation Learning

Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

no code implementations SEMEVAL 2019 Najoung Kim, Roma Patel, Adam Poliak, Alex Wang, Patrick Xia, R. Thomas McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, Ellie Pavlick

Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably.

CCG Supertagging Language Modelling +1

Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop

no code implementations5 Apr 2019 Afra Alishahi, Grzegorz Chrupała, Tal Linzen

The EMNLP 2018 workshop BlackboxNLP was dedicated to resources and techniques specifically developed for analyzing and understanding the inner-workings and representations acquired by neural models of language.

Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages

2 code implementations NAACL 2019 Shauli Ravfogel, Yoav Goldberg, Tal Linzen

How do typological properties such as word order and morphological case marking affect the ability of neural sequence models to acquire the syntax of a language?

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

5 code implementations ACL 2019 R. Thomas McCoy, Ellie Pavlick, Tal Linzen

We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics.

Natural Language Inference

Human few-shot learning of compositional instructions

2 code implementations14 Jan 2019 Brenden M. Lake, Tal Linzen, Marco Baroni

There have been striking recent improvements in machine learning for natural language processing, yet the best algorithms require vast amounts of experience and struggle to generalize new concepts in compositional ways.

Few-Shot Learning

RNNs Implicitly Implement Tensor Product Representations

no code implementations20 Dec 2018 R. Thomas McCoy, Tal Linzen, Ewan Dunbar, Paul Smolensky

Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).

Representation Learning

Non-entailed subsequences as a challenge for natural language inference

no code implementations29 Nov 2018 R. Thomas McCoy, Tal Linzen

Neural network models have shown great success at natural language inference (NLI), the task of determining whether a premise entails a hypothesis.

Natural Language Inference

Can Entropy Explain Successor Surprisal Effects in Reading?

no code implementations WS 2019 Marten van Schijndel, Tal Linzen

Human reading behavior is sensitive to surprisal: more predictable words tend to be read faster.

Language Modelling

What can linguistics and deep learning contribute to each other?

no code implementations11 Sep 2018 Tal Linzen

Joe Pater's target article calls for greater interaction between neural network research and linguistics.

Language Acquisition

A Neural Model of Adaptation in Reading

1 code implementation EMNLP 2018 Marten van Schijndel, Tal Linzen

It has been argued that humans rapidly adapt their lexical and syntactic expectations to match the statistics of the current linguistic context.

Language Modelling

Targeted Syntactic Evaluation of Language Models

5 code implementations EMNLP 2018 Rebecca Marvin, Tal Linzen

We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence.

CCG Supertagging Language Modelling

Distinct patterns of syntactic agreement errors in recurrent networks and humans

1 code implementation18 Jul 2018 Tal Linzen, Brian Leonard

To examine the extent to which the syntactic representations of these networks are similar to those used by humans when processing sentences, we compare the detailed pattern of errors that RNNs and humans make on this task.

Colorless green recurrent networks dream hierarchically

2 code implementations NAACL 2018 Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, Marco Baroni

Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language.

Language Modelling

Exploring the Syntactic Abilities of RNNs with Multi-task Learning

1 code implementation CONLL 2017 Emile Enguehard, Yoav Goldberg, Tal Linzen

Recent work has explored the syntactic abilities of RNNs using the subject-verb agreement task, which diagnoses sensitivity to sentence structure.

CCG Supertagging Language Modelling +1

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

5 code implementations TACL 2016 Tal Linzen, Emmanuel Dupoux, Yoav Goldberg

The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities.

Language Modelling

Issues in evaluating semantic spaces using word analogies

no code implementations WS 2016 Tal Linzen

The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: it is considered desirable for a space to represent semantic relations as consistent vector offsets.

Cannot find the paper you are looking for? You can Submit a new open access paper.