Search Results for author:
Found 75 papers, 40 papers with code
How Does Code Pretraining Affect Language Model Task Performance?
Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code.
Testing learning hypotheses using neural networks by manipulating learning data
We then show that a neural network language model can learn restrictions to the passive that are similar to those displayed by humans, suggesting that evidence for these exceptions is available in the linguistic input.
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques.
SPAWNing Structural Priming Predictions from a Cognitively Motivated Parser
Structural priming is a widely used psycholinguistic paradigm to study human sentence representations.
Can You Learn Semantics Through Next-Word Prediction? The Case of Entailment
Do LMs infer the semantics of text from co-occurrence patterns in their training data?
In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks: given labeled examples in the input context, the LLM learns to perform the task without weight updates.
A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises.
The Impact of Depth on Compositional Generalization in Transformer Language Models
Because model latency is approximately linear in the number of layers, these results lead us to the recommendation that, with a given total parameter budget, transformers can be made shallower than is typical without sacrificing performance.
A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing
A recent attempt to create a unified cognitive model integrating these two factors relied on the parallels between the self-attention mechanism of transformer language models and cue-based retrieval theories of working memory in human sentence processing (Ryu and Lewis 2021).
SLOG: A Structural Generalization Benchmark for Semantic Parsing
The goal of compositional generalization benchmarks is to evaluate how well models generalize to new complex linguistic expressions.
Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number
Deep architectures such as Transformers are sometimes criticized for having uninterpretable "black-box" representations.
Do Language Models' Words Refer?
There is prima facie reason to think they do not since LMs do not interact with the world in the way that ordinary language users do.
Language Models Can Learn Exceptions to Syntactic Rules
Artificial neural networks can generalize productively to novel contexts.
How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases
Accurate syntactic representations are essential for robust generalization in natural language.
How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech
When acquiring syntax, children consistently choose hierarchical rules over competing non-hierarchical possibilities.
Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models
Human linguistic capacity is often characterized by compositionality and the generalization it enables -- human learners can produce and comprehend novel complex expressions by composing known parts.
Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models
However, much of this analysis has focused on monolingual models, and analyses of multilingual models have employed correlational methods that are confounded by the choice of probing tasks.
Characterizing Verbatim Short-Term Memory in Neural Language Models
We tested whether language models could retrieve the exact words that occurred previously in a text.
Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities
We find that treating syntactic predictability independently from lexical predictability indeed results in larger estimates of garden path.
Entailment Semantics Can Be Extracted from an Ideal Language Model
Language models are often trained on text alone, without additional grounding.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it
We use this evaluation suite for a fine-grained investigation of the entity tracking abilities of the Transformer-based models GPT-2 and GPT-3.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We find that pre-trained seq2seq models generalize hierarchically when performing syntactic transformations, whereas models trained from scratch on syntactic transformations do not.
Improving Compositional Generalization with Latent Structure and Data Augmentation
Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization.
How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN
We apply these analyses to four neural language models (an LSTM, a Transformer, Transformer-XL, and GPT-2).
Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks
Neural network models often generalize poorly to mismatched domains or distributions.
Short-term memory in neural language models
Thus, although the transformer and LSTM architectures were both trained to predict language sequences, only the transformer learned to flexibly index prior tokens.
The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generation
Temporary syntactic ambiguities arise when the beginning of a sentence is compatible with multiple syntactic analyses.
Frequency Effects on Syntactic Rule Learning in Transformers
Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules.
NOPE: A Corpus of Naturally-Occurring Presuppositions in English
Understanding language requires grasping not only the overtly stated content, but also making inferences about things that were left unsaid.
The MultiBERTs: BERT Reproductions for Robustness Analysis
Experiments with pre-trained models such as BERT are often based on a single checkpoint.
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models
Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts.
Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction
We apply this method to study how BERT models of different sizes process relative clauses (RCs).
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark
To this end, we introduce the Benchmark for Evaluation of Grounded INteraction (BEGIN), comprised of 12k dialogue turns generated by neural dialogue systems trained on three knowledge-grounded dialogue corpora.
Does Putting a Linguist in the Loop Improve NLU Data Collection?
We take natural language inference as a test case and ask whether it is beneficial to put a linguist `in the loop' during data collection to dynamically identify and address gaps in the data by introducing novel constraints on the task.
Information-theoretic Probing Explains Reliance on Spurious Features
In this work, we test the hypothesis that the extent to which a feature influences a model's decisions can be predicted using a combination of two factors: The feature's "extractability" after pre-training (measured using information-theoretic probing techniques), and the "evidence" available during fine-tuning (defined as the feature's co-occurrence rate with the label).
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts.
Universal linguistic inductive biases via meta-learning
To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases.
How Can We Accelerate Progress Towards Human-like Linguistic Generalization?
This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has become a central tool for measuring progress in natural language understanding.
Cross-Linguistic Syntactic Evaluation of Word Prediction Models
On other constructions, agreement accuracy was generally higher in languages with richer morphology.
Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs
Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks.
Syntactic Data Augmentation Increases Robustness to Inference Heuristics
Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets.
Syntactic Structure from Deep Learning
Modern deep neural networks achieve impressive performance in engineering applications that require extensive linguistic skills, such as machine translation.
Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks
We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection.
BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance
If the same neural network architecture is trained multiple times on the same dataset, will it make similar linguistic generalizations across runs?
Discovering the Compositional Structure of Vector Representations with Role Learning Networks
How can neural networks perform so well on compositional tasks even though they lack explicit compositional representations?
Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Models
Neural language models (LMs) perform well on tasks that require sensitivity to syntactic structure.
Quantity doesn't buy quality syntax with neural language models
We investigate to what extent these shortcomings can be mitigated by increasing the size of the network and the corpus on which it is trained.
RNNs implicitly implement tensor-product representations
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably.
Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop
The EMNLP 2018 workshop BlackboxNLP was dedicated to resources and techniques specifically developed for analyzing and understanding the inner-workings and representations acquired by neural models of language.
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages
How do typological properties such as word order and morphological case marking affect the ability of neural sequence models to acquire the syntax of a language?
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics.
Human few-shot learning of compositional instructions
There have been striking recent improvements in machine learning for natural language processing, yet the best algorithms require vast amounts of experience and struggle to generalize new concepts in compositional ways.
RNNs Implicitly Implement Tensor Product Representations
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).
Non-entailed subsequences as a challenge for natural language inference
Neural network models have shown great success at natural language inference (NLI), the task of determining whether a premise entails a hypothesis.
Can Entropy Explain Successor Surprisal Effects in Reading?
Human reading behavior is sensitive to surprisal: more predictable words tend to be read faster.
What can linguistics and deep learning contribute to each other?
Joe Pater's target article calls for greater interaction between neural network research and linguistics.
A Neural Model of Adaptation in Reading
It has been argued that humans rapidly adapt their lexical and syntactic expectations to match the statistics of the current linguistic context.
Targeted Syntactic Evaluation of Language Models
We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence.
Distinct patterns of syntactic agreement errors in recurrent networks and humans
To examine the extent to which the syntactic representations of these networks are similar to those used by humans when processing sentences, we compare the detailed pattern of errors that RNNs and humans make on this task.
Colorless green recurrent networks dream hierarchically
Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
We examine this proposal using recurrent neural networks (RNNs), which are not constrained in such a way.
Phonological (un)certainty weights lexical activation
Spoken word recognition involves at least two basic computations.
Exploring the Syntactic Abilities of RNNs with Multi-task Learning
Recent work has explored the syntactic abilities of RNNs using the subject-verb agreement task, which diagnoses sensitivity to sentence structure.
Comparing Character-level Neural Language Models Using a Lexical Decision Task
What is the information captured by neural network models of language?
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities.
Issues in evaluating semantic spaces using word analogies
The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: it is considered desirable for a space to represent semantic relations as consistent vector offsets.
