Search Results for author:
Found 15 papers, 6 papers with code
RNNs can generate bounded hierarchical languages with optimal memory
Recurrent neural networks empirically generate natural language with high syntactic fidelity.
A Cross-Linguistic Pressure for Uniform Information Density in Word Order
Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically.
Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text
Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks.
An Information-Theoretic Characterization of Morphological Fusion
Linguistic typology generally divides synthetic languages into groups based on their morphological fusion.
Modeling Human Reading with Neural Attention
When humans read text, they fixate some words and skip others.
Modeling Task Effects in Human Reading with Neural Network-based Attention
Research on human reading has long documented that reading behavior shows task-specific effects, but it has been challenging to build general models predicting what reading behavior humans will show in a given task.
Character-based Surprisal as a Model of Reading Difficulty in the Presence of Error
Also, transpositions are more difficult than misspellings, and a high error rate increases difficulty for all words, including correct ones.
Theoretical Limitations of Self-Attention in Neural Sequence Models
These limitations seem surprising given the practical success of self-attention and the prominent role assigned to hierarchical structure in linguistics, suggesting that natural language can be approximated well with models that are too weak for the formal languages typically assumed in theoretical linguistics.
Sensitivity as a Complexity Measure for Sequence Classification Tasks
We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity.
Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality
Using data from 80 languages in 17 language families and phylogenetic modeling, we demonstrate that languages evolve to balance these pressures, such that word order change is accompanied by change in the frequency distribution of the syntactic structures which speakers communicate to maintain overall efficiency.
A Theory of Emergent In-Context Learning as Implicit Structure Induction
Scaling large language models (LLMs) leads to an emergent capacity to learn in-context from example demonstrations.
Why are Sensitive Functions Hard for Transformers?
We show theoretically and empirically that this theory unifies a broad array of empirical observations about the learning abilities and biases of transformers, such as their generalization bias towards low sensitivity and low degree, and difficulty in length generalization for PARITY.
