Search Results for author:
Found 36 papers, 12 papers with code
Learning to Reason via Mixture-of-Thought for Logical Reasoning
To fill in this gap, we propose Mixture-of-Thought (MoT), a framework that enables LLMs to reason across three complementary modalities: natural language, code, and a newly introduced symbolic modality, truth-table, which systematically enumerates logical cases and partially mitigates key failure modes in natural language reasoning.
Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models
Accuracy remains a standard metric for evaluating AI systems, but it offers limited insight into how models arrive at their solutions.
Identifying and Mitigating the Influence of the Prior Distribution in Large Language Models
Specifically, we show that it is possible to identify layers of the underlying neural network that correlate with the prior probability of a response and that lightweight finetuning of these layers with basic prompts on prior-dominated tasks achieves high performance on held-out answers.
Convolutional Neural Networks Can (Meta-)Learn the Same-Different Relation
While convolutional neural networks (CNNs) have come to match and exceed human performance in many settings, the tasks these models optimize for are largely constrained to the level of individual objects, such as classification and captioning.
Teasing Apart Architecture and Initial Weights as Sources of Inductive Bias in Neural Networks
But what those networks can learn depends upon their inductive biases -- the factors other than the data that influence the solutions they discover -- and the inductive biases of neural networks remain poorly understood, limiting our ability to draw conclusions about human learning from the performance of these systems.
Minimization of Boolean Complexity in In-Context Concept Learning
What factors contribute to the relative success and corresponding difficulties of in-context learning for Large Language Models (LLMs)?
When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1
In "Embers of Autoregression" (McCoy et al., 2023), we showed that several large language models (LLMs) have some important limitations that are attributable to their origins in next-word prediction.
Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
By focusing on a single relatively simple task, we are able to identify three factors that systematically affect CoT performance: the probability of the task's expected output (probability), what the model has implicitly learned during pre-training (memorization), and the number of intermediate operations involved in reasoning (noisy reasoning).
Is In-Context Learning a Type of Error-Driven Learning? Evidence from the Inverse Frequency Effect in Structural Priming
In our experiments, we simulated structural priming with ICL and found that LLMs indeed display the IFE, with the effect being stronger in larger models.
modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models
We introduce modeLing, a novel benchmark of Linguistics Olympiad-style puzzles which tests few-shot reasoning in AI systems.
Distilling Symbolic Priors for Concept Learning into Neural Networks
Is it possible to create a neural network that displays the same inductive biases?
Deep de Finetti: Recovering Topic Distributions from Large Language Models
Large language models (LLMs) can produce long, coherent passages of text, suggesting that LLMs, although trained on next-word prediction, must represent the latent structure that characterizes a document.
Bayes in the age of intelligent machines
The success of methods based on artificial neural networks in creating intelligent machines seems like it might pose a challenge to explanations of human cognition in terms of Bayesian inference.
Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
This approach - which we call the teleological approach - leads us to identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input.
Modeling rapid language learning by distilling Bayesian priors into artificial neural networks
We show that learning from limited naturalistic data is possible with an approach that combines the strong inductive biases of a Bayesian model with the flexible representations of a neural network.
How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech
When acquiring syntax, children consistently choose hierarchical rules over competing non-hierarchical possibilities.
Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages
Machine translation has seen rapid progress with the advent of Transformer-based models.
Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems
What explains the dramatic progress from 20th-century to 21st-century AI, and how can the remaining limitations of current AI be overcome?
How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN
We apply these analyses to four neural language models (an LSTM, a Transformer, Transformer-XL, and GPT-2).
Picking BERT's Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis
As the name implies, contextualized representations of language are typically motivated by their ability to encode context.
Universal linguistic inductive biases via meta-learning
To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases.
Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs
Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks.
Syntactic Data Augmentation Increases Robustness to Inference Heuristics
Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets.
Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks
We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection.
BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance
If the same neural network architecture is trained multiple times on the same dataset, will it make similar linguistic generalizations across runs?
What do you learn from context? Probing for sentence structure in contextualized word representations
The jiant toolkit for general-purpose text understanding models
RNNs implicitly implement tensor-product representations
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling
Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo (Peters et al., 2018).
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably.
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics.
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
Natural language understanding has recently seen a surge of progress with the use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which are pretrained on variants of language modeling.
RNNs Implicitly Implement Tensor Product Representations
Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).
Non-entailed subsequences as a challenge for natural language inference
Neural network models have shown great success at natural language inference (NLI), the task of determining whether a premise entails a hypothesis.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
We examine this proposal using recurrent neural networks (RNNs), which are not constrained in such a way.
