Search Results for author:
Found 23 papers, 10 papers with code
An Information-Theoretic Characterization of Morphological Fusion
Linguistic typology generally divides synthetic languages into groups based on their morphological fusion.
One Size Fits None: Rethinking Fairness in Medical AI
Machine learning (ML) models are increasingly used to support clinical decision-making.
Born a Transformer -- Always a Transformer?
Mechanistic analysis reveals that this asymmetry is connected to the differences in the strength of induction versus anti-induction circuits within pretrained transformers.
Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers
Chain-of-thought reasoning and scratchpads have emerged as critical tools for enhancing the computational capabilities of transformers.
A Formal Framework for Understanding Length Generalization in Transformers
A major challenge for transformers is generalizing to sequences longer than those observed during training.
Separations in the Representational Capabilities of Transformers and Recurrent Architectures
Furthermore, we show that two-layer Transformers of logarithmic size can perform decision tasks such as string equality or disjointness, whereas both one-layer Transformers and recurrent models require linear size for these tasks.
InversionView: A General-Purpose Method for Reading Information from Neural Activations
The inner workings of neural networks can be better understood if we can fully decipher the information encoded in neural activations.
The Expressive Capacity of State Space Models: A Formal Language Perspective
Recently, recurrent models based on linear state space models (SSMs) have shown promising performance in language modeling (LM), competititve with transformers.
Linguistic Structure from a Bottleneck on Sequential Information Processing
It establishes a link between the statistical and algebraic structure of human language, and reinforces the idea that the structure of human language is shaped by communication under cognitive constraints.
Why are Sensitive Functions Hard for Transformers?
We show theoretically and empirically that this theory unifies a broad array of empirical observations about the learning abilities and biases of transformers, such as their generalization bias towards low sensitivity and low degree, and difficulty in length generalization for PARITY.
A Cross-Linguistic Pressure for Uniform Information Density in Word Order
Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically.
A Theory of Emergent In-Context Learning as Implicit Structure Induction
Scaling large language models (LLMs) leads to an emergent capacity to learn in-context from example demonstrations.
Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality
Using data from 80 languages in 17 language families and phylogenetic modeling, we demonstrate that languages evolve to balance these pressures, such that word order change is accompanied by change in the frequency distribution of the syntactic structures which speakers communicate to maintain overall efficiency.
Sensitivity as a Complexity Measure for Sequence Classification Tasks
We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity.
RNNs can generate bounded hierarchical languages with optimal memory
Recurrent neural networks empirically generate natural language with high syntactic fidelity.
Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text
Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks.
Theoretical Limitations of Self-Attention in Neural Sequence Models
These limitations seem surprising given the practical success of self-attention and the prominent role assigned to hierarchical structure in linguistics, suggesting that natural language can be approximated well with models that are too weak for the formal languages typically assumed in theoretical linguistics.
Character-based Surprisal as a Model of Reading Difficulty in the Presence of Error
Also, transpositions are more difficult than misspellings, and a high error rate increases difficulty for all words, including correct ones.
Modeling Task Effects in Human Reading with Neural Network-based Attention
Research on human reading has long documented that reading behavior shows task-specific effects, but it has been challenging to build general models predicting what reading behavior humans will show in a given task.
Modeling Human Reading with Neural Attention
When humans read text, they fixate some words and skip others.
