1 code implementation • 6 May 2022 • Sebastian Schuster, Tal Linzen
We use this evaluation suite for a fine-grained investigation of the entity tracking abilities of the Transformer-based models GPT-2 and GPT-3.
1 code implementation • Findings (ACL) 2022 • Aaron Mueller, Robert Frank, Tal Linzen, Luheng Wang, Sebastian Schuster
We find that pre-trained seq2seq models generalize hierarchically when performing syntactic transformations, whereas models trained from scratch on syntactic transformations do not.
1 code implementation • 14 Dec 2021 • Linlu Qiu, Peter Shaw, Panupong Pasupat, Paweł Krzysztof Nowak, Tal Linzen, Fei Sha, Kristina Toutanova
Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization.
no code implementations • 18 Nov 2021 • R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, Asli Celikyilmaz
We apply these analyses to four neural language models (an LSTM, a Transformer, Transformer-XL, and GPT-2).
1 code implementation • 9 Nov 2021 • Wang Zhu, Peter Shaw, Tal Linzen, Fei Sha
Neural network models often generalize poorly to mismatched domains or distributions.
no code implementations • 29 Sep 2021 • Kristijan Armeni, Christopher Honey, Tal Linzen
Thus, although the transformer and LSTM architectures were both trained to predict language sequences, only the transformer learned to flexibly index prior tokens.
no code implementations • EMNLP (BlackboxNLP) 2021 • Laura Aina, Tal Linzen
Temporary syntactic ambiguities arise when the beginning of a sentence is compatible with multiple syntactic analyses.
1 code implementation • CoNLL (EMNLP) 2021 • Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bowman, Tal Linzen
Understanding language requires grasping not only the overtly stated content, but also making inferences about things that were left unsaid.
1 code implementation • EMNLP 2021 • Jason Wei, Dan Garrette, Tal Linzen, Ellie Pavlick
Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules.
1 code implementation • ICLR 2022 • Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, Ian Tenney, Ellie Pavlick
Experiments with pre-trained models such as BERT are often based on a single checkpoint.
1 code implementation • ACL 2021 • Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, Yonatan Belinkov
Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts.
no code implementations • CoNLL (EMNLP) 2021 • Shauli Ravfogel, Grusha Prasad, Tal Linzen, Yoav Goldberg
We apply this method to study how BERT models of different sizes process relative clauses (RCs).
no code implementations • 30 Apr 2021 • Nouha Dziri, Hannah Rashkin, Tal Linzen, David Reitter
To facilitate evaluation of such metrics, we introduce the Benchmark for Evaluation of Grounded INteraction (BEGIN).
no code implementations • Findings (EMNLP) 2021 • Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alex Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, Samuel R. Bowman
We take natural language inference as a test case and ask whether it is beneficial to put a linguist `in the loop' during data collection to dynamically identify and address gaps in the data by introducing novel constraints on the task.
no code implementations • ICLR 2021 • Charles Lovering, Rohan Jha, Tal Linzen, Ellie Pavlick
In this work, we test the hypothesis that the extent to which a feature influences a model's decisions can be predicted using a combination of two factors: The feature's "extractability" after pre-training (measured using information-theoretic probing techniques), and the "evidence" available during fine-tuning (defined as the feature's co-occurrence rate with the label).
1 code implementation • EMNLP 2020 • Najoung Kim, Tal Linzen
Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts.
1 code implementation • 29 Jun 2020 • R. Thomas McCoy, Erin Grant, Paul Smolensky, Thomas L. Griffiths, Tal Linzen
To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases.
no code implementations • ACL 2020 • Tal Linzen
This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has become a central tool for measuring progress in natural language understanding.
2 code implementations • ACL 2020 • Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, Tal Linzen
On other constructions, agreement accuracy was generally higher in languages with richer morphology.
1 code implementation • ACL 2020 • Michael A. Lepori, Tal Linzen, R. Thomas McCoy
Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks.
1 code implementation • ACL 2020 • Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, Tal Linzen
Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets.
no code implementations • 22 Apr 2020 • Tal Linzen, Marco Baroni
Modern deep neural networks achieve impressive performance in engineering applications that require extensive linguistic skills, such as machine translation.
no code implementations • TACL 2020 • R. Thomas McCoy, Robert Frank, Tal Linzen
We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection.
no code implementations • EMNLP (BlackboxNLP) 2020 • R. Thomas McCoy, Junghyun Min, Tal Linzen
If the same neural network architecture is trained multiple times on the same dataset, will it make similar linguistic generalizations across runs?
2 code implementations • EMNLP (BlackboxNLP) 2020 • Paul Soulos, Tom McCoy, Tal Linzen, Paul Smolensky
How can neural networks perform so well on compositional tasks even though they lack explicit compositional representations?
1 code implementation • CONLL 2019 • Grusha Prasad, Marten Van Schijndel, Tal Linzen
Neural language models (LMs) perform well on tasks that require sensitivity to syntactic structure.
no code implementations • IJCNLP 2019 • Marten van Schijndel, Aaron Mueller, Tal Linzen
We investigate to what extent these shortcomings can be mitigated by increasing the size of the network and the corpus on which it is trained.
1 code implementation • ICLR 2019 • R. Thomas McCoy, Tal Linzen, Ewan Dunbar, Paul Smolensky
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).
no code implementations • SEMEVAL 2019 • Najoung Kim, Roma Patel, Adam Poliak, Alex Wang, Patrick Xia, R. Thomas McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, Ellie Pavlick
Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably.
no code implementations • 5 Apr 2019 • Afra Alishahi, Grzegorz Chrupała, Tal Linzen
The EMNLP 2018 workshop BlackboxNLP was dedicated to resources and techniques specifically developed for analyzing and understanding the inner-workings and representations acquired by neural models of language.
2 code implementations • NAACL 2019 • Shauli Ravfogel, Yoav Goldberg, Tal Linzen
How do typological properties such as word order and morphological case marking affect the ability of neural sequence models to acquire the syntax of a language?
5 code implementations • ACL 2019 • R. Thomas McCoy, Ellie Pavlick, Tal Linzen
We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics.
2 code implementations • 14 Jan 2019 • Brenden M. Lake, Tal Linzen, Marco Baroni
There have been striking recent improvements in machine learning for natural language processing, yet the best algorithms require vast amounts of experience and struggle to generalize new concepts in compositional ways.
no code implementations • 20 Dec 2018 • R. Thomas McCoy, Tal Linzen, Ewan Dunbar, Paul Smolensky
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).
no code implementations • 29 Nov 2018 • R. Thomas McCoy, Tal Linzen
Neural network models have shown great success at natural language inference (NLI), the task of determining whether a premise entails a hypothesis.
no code implementations • WS 2019 • Marten van Schijndel, Tal Linzen
Human reading behavior is sensitive to surprisal: more predictable words tend to be read faster.
no code implementations • 11 Sep 2018 • Tal Linzen
Joe Pater's target article calls for greater interaction between neural network research and linguistics.
1 code implementation • EMNLP 2018 • Marten van Schijndel, Tal Linzen
It has been argued that humans rapidly adapt their lexical and syntactic expectations to match the statistics of the current linguistic context.
5 code implementations • EMNLP 2018 • Rebecca Marvin, Tal Linzen
We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence.
1 code implementation • 18 Jul 2018 • Tal Linzen, Brian Leonard
To examine the extent to which the syntactic representations of these networks are similar to those used by humans when processing sentences, we compare the detailed pattern of errors that RNNs and humans make on this task.
2 code implementations • NAACL 2018 • Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, Marco Baroni
Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language.
no code implementations • 25 Feb 2018 • R. Thomas McCoy, Robert Frank, Tal Linzen
We examine this proposal using recurrent neural networks (RNNs), which are not constrained in such a way.
no code implementations • WS 2018 • Laura Gwilliams, David Poeppel, Alec Marantz, Tal Linzen
Spoken word recognition involves at least two basic computations.
1 code implementation • CONLL 2017 • Emile Enguehard, Yoav Goldberg, Tal Linzen
Recent work has explored the syntactic abilities of RNNs using the subject-verb agreement task, which diagnoses sensitivity to sentence structure.
no code implementations • EACL 2017 • Ga{\"e}l Le Godais, Tal Linzen, Emmanuel Dupoux
What is the information captured by neural network models of language?
5 code implementations • TACL 2016 • Tal Linzen, Emmanuel Dupoux, Yoav Goldberg
The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities.
no code implementations • WS 2016 • Tal Linzen
The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: it is considered desirable for a space to represent semantic relations as consistent vector offsets.