Search Results for author:
Found 26 papers, 10 papers with code
Distilling Symbolic Priors for Concept Learning into Neural Networks
Is it possible to create a neural network that displays the same inductive biases?
Deep de Finetti: Recovering Topic Distributions from Large Language Models
Large language models (LLMs) can produce long, coherent passages of text, suggesting that LLMs, although trained on next-word prediction, must represent the latent structure that characterizes a document.
Bayes in the age of intelligent machines
The success of methods based on artificial neural networks in creating intelligent machines seems like it might pose a challenge to explanations of human cognition in terms of Bayesian inference.
Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
This approach - which we call the teleological approach - leads us to identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input.
Modeling rapid language learning by distilling Bayesian priors into artificial neural networks
We show that learning from limited naturalistic data is possible with an approach that combines the strong inductive biases of a Bayesian model with the flexible representations of a neural network.
How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech
When acquiring syntax, children consistently choose hierarchical rules over competing non-hierarchical possibilities.
Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages
Machine translation has seen rapid progress with the advent of Transformer-based models.
Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems
What explains the dramatic progress from 20th-century to 21st-century AI, and how can the remaining limitations of current AI be overcome?
How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN
We apply these analyses to four neural language models (an LSTM, a Transformer, Transformer-XL, and GPT-2).
Picking BERT's Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis
As the name implies, contextualized representations of language are typically motivated by their ability to encode context.
Universal linguistic inductive biases via meta-learning
To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases.
Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs
Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks.
Syntactic Data Augmentation Increases Robustness to Inference Heuristics
Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets.
Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks
We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection.
BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance
If the same neural network architecture is trained multiple times on the same dataset, will it make similar linguistic generalizations across runs?
What do you learn from context? Probing for sentence structure in contextualized word representations
The jiant toolkit for general-purpose text understanding models
RNNs implicitly implement tensor-product representations
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling
Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo (Peters et al., 2018).
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably.
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics.
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
Natural language understanding has recently seen a surge of progress with the use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which are pretrained on variants of language modeling.
RNNs Implicitly Implement Tensor Product Representations
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).
Non-entailed subsequences as a challenge for natural language inference
Neural network models have shown great success at natural language inference (NLI), the task of determining whether a premise entails a hypothesis.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
We examine this proposal using recurrent neural networks (RNNs), which are not constrained in such a way.
